I used tempfile.mkdtemp() to create a temporary directory for my runs (as I don’t want a persistent folder with tons of runs)
For training everything works fine but when resuming the run to do some validation / evaluation updates, and using run.summary.update({"key": value}) I got a
wandb: WARNING Path /tmp/tmpq5uafy4d/wandb/ wasn't writable, using system temp directory
with obviously
File "/mnt/Projets/nlp/.venv/lib/python3.9/site-packages/wandb/sdk/internal/sender.py", line 855, in _update_summary
with open(summary_path, "w") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpq5uafy4d/wandb/run-20211102_153311-37264m5k/files/wandb-summary.json'
The directory is readable, writable, and searchable only by the creating user ID.
So I guess WandB is not using the user ID and thus is not able to write in the directory for updating.
Note that this directory is different from the training one (as it’s random at each init)
After searching for a long time with it, wandb firstly deletes my class object (which call deletion of the temp folder) and then try to update the run.
To avoid that you need to first call run.finish() or wandb.finish() which will first update and then delete your object (or let the garbage collector do it) thus it will be synced before the object is destroyed and the temp file removed.
Hi @ierezell, there are some reasons why you don’t want to automatically delete your run folders, especially if there is an issue with the run. However, I do understand your desire to manage the clutter in your file system. It sounds like having a feature where you could tell wandb to delete the local files it created after a successful run would be the most preferable option for you. Currently, calling wandb sync --clean will sync any unsynced runs and then remove those run folders from your computer. The wandb dir will still be there and there but it will only be taking up bytes of space as most of the information will have been deleted. If you still find it annoying to have mostly empty ./wandb folders in your project dirs, you can set the WANDB_DIR environment variable to an absolute path where all of your run data will be stored until you call wandb sync --clean.
I didn’t know the wandb sync --clean option!
It’s quite what I wanted to do, saving space and your solution will only delete finished runs which is nice
I don’t mind deleting failed run as I can just relaunch them again (it’s small models on only one machine). This is why I put all my wandb folders in /tmp which means I keep the folder until I reboot.
Is there any option to have the clean feature enabled by default? I mean for any run if sync is complete: delete the folder.
Thanks for the response, I guess it solves it but we can continue discussing it.
As of now, there isn’t an option to enable that by default. One issue you should be aware of is that if you are currently logging a run when you call wandb sync --clean bad things will happen. We’re working on improving the the robustness of these features and will likely support an automatic clean option in the future.
