Images logged using W&B logger bloats up /tmp

In my code, I have the following statement called at the end of every epoch as part of validation

wandb_logger.experiment.log({'test': wandb.Image(img)}, commit=commit)

where img is an image of type numpy array.

I have noticed that W&B logger writes the image to a directory inside /tmp, and this directory is only cleared at the end of the run.

I have limited space in /tmp and this leads to my training run crashing. I have tried saving the image to disk and then calling W&B log on the path, this skips saving to /tmp but doesn’t really work well with distributed training (deadlock issues).

Expected behavior:
The user should be able to configure the temporary directory where intermediary media is stored.

Kindly let me know if there’s any other workaround.

Hi @suryatejrmatician , happy to help. Could you try changing g your cache directory by setting the WANDB_CACHE_DIR environment variable. As we write artifacts through the cache, after updating this directory, wandb will update where intermediary files are stored prior to logging them.