Images logged using W&B logger bloats up /tmp

W&B Help
1

In my code, I have the following statement called at the end of every epoch as part of validation

wandb_logger.experiment.log({'test': wandb.Image(img)}, commit=commit)

where img is an image of type numpy array.

I have noticed that W&B logger writes the image to a directory inside /tmp, and this directory is only cleared at the end of the run.

I have limited space in /tmp and this leads to my training run crashing. I have tried saving the image to disk and then calling W&B log on the path, this skips saving to /tmp but doesn’t really work well with distributed training (deadlock issues).

Expected behavior:
The user should be able to configure the temporary directory where intermediary media is stored.

Kindly let me know if there’s any other workaround.

2

Hi @suryatejrmatician , happy to help. Could you try changing g your cache directory by setting the WANDB_CACHE_DIR environment variable. As we write artifacts through the cache, after updating this directory, wandb will update where intermediary files are stored prior to logging them.

3

Hi @suryatejrmatician since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!

4

This sounds like a similar problem that my team was facing. My artifacts were being logged into the WandB account. I wrote a small gist which could tackle it. Its a kind of a workaround, let me know if it helps:

5

Hey @mohammadbakir, I have tried this and it doesn’t seem to be working. The temporary files are still logged to /tmp, note that this is also for all logging that w&b does, not necessaily the image logging example that I have given above.

6

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.