I have noticed that W&B logger writes the image to a directory inside /tmp, and this directory is only cleared at the end of the run.
I have limited space in /tmp and this leads to my training run crashing. I have tried saving the image to disk and then calling W&B log on the path, this skips saving to /tmp but doesn’t really work well with distributed training (deadlock issues).
The user should be able to configure the temporary directory where intermediary media is stored.
Kindly let me know if there’s any other workaround.
Hi @suryatejrmatician , happy to help. Could you try changing g your cache directory by setting the WANDB_CACHE_DIR environment variable. As we write artifacts through the cache, after updating this directory, wandb will update where intermediary files are stored prior to logging them.
This sounds like a similar problem that my team was facing. My artifacts were being logged into the WandB account. I wrote a small gist which could tackle it. Its a kind of a workaround, let me know if it helps:
Hey @mohammadbakir, I have tried this and it doesn’t seem to be working. The temporary files are still logged to /tmp, note that this is also for all logging that w&b does, not necessaily the image logging example that I have given above.