I am having an issue of running wandb on a slurm cluster and the artifacts populate in my user dir ~/.local/share/wandb/artifacts/staging. This dir is populated with tmp* files. I want to change the environment variable to reroute this path, but none of the paths listed on the storage docs do it.
Those same docs say that you can run$ wandb artifact cache cleanup 1GB from the cmd line but it is unclear of this is for the cache ~/.cache. Since this command only caps the cache it is hard to tell if it is working without hitting the 1GB limit. Can you help clarify if this command is for ~/.local/share/wandb/artifacts/staging or if it is for ~/.cache? Thanks for any help.
Hi @mjvolk3, thanks for your question! Do you know if .local/share is the system cache directory? The default path should be ~/cache/wandb/artifacts/ unless you set WANDB_CACHE_DIR. Said this, wandb artifact cache cleanup should be for ~/cache/wandb/artifacts/ since this is the default cache dir. Also, could you check with env | grep -i wandb* if WANDB_CACHE_DIR is set?
We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.
Hi there, I do think that .local/share is the system cache directory. This is a bit of an issue because we have limited space on our login node to 25 GB and ~/cache/wandb/artifacts/staging will often go over my memory capacity. Within this dir there are many files like tmp4xngbfu5 , which I assume is some temporary file for uploading artifacts to wandb server. This does seem to be distinct from WANDB_CACHE_DIR because I have started manually setting this to a different directory within my python scripts. For now I just delete this dir when it gets too large but this is not a sustainable solution.