I run wandb offline and then sync the runs to the dashboard using wandb sync. Now, I also actively delete some runs from the dashboard, so as to keep my workspace clean. The issue is, when I run the sync command locally, it keeps trying to sync deleted runs for a long time before moving on to the next folder. For example, it will try almost 7 times on the same name before it moves on to sync the next folder. This makes the sync process very slow. Is there a way around this so that I can automatically delete the folders of experiments that are already purged in the dashboard? Or perhaps, somehow skip/reduce the number of tries per folder while syncing?
I tried the --clean flag, but it does not find these runs. It says there are no runs in the past 24 hours. I also tried reducing the hours using the flag --clean-old-hours but it does not detect these runs.
This is the error that the command prints when it tries to sync a deleted run: wandb: ERROR Error while calling W&B API: run <nur_name> was previously created and deleted; try a new run name (<Response [409]>)
wandb sync uploads an offline training directory to W&B, so deleting it in the UI does not necessarily guarantee that it won’t be synced when you call the command.
I am trying to reproduce the behavior on my end by:
Syncing some offline runs
Deleting one run in my workspace
Running another offline run
Syncing the runs again
I was unable to reproduce this behavior with the above workflow, so please fill me in if I am misunderstanding the steps you’ve outlined.
When you run wandb sync prior to running wandb sync --sync-all do you see the correct number of files to be synced? Additionally, what SDK version are you on?
I believe this issue will be reproducible only if you also choose to delete the artifact that was created along with the experiment while deleting the experiment from the WandB UI. I have shared the error I see while syncing: wandb: ERROR Error while calling W&B API: run <my_deleted_run> was previously created and deleted; try a new run name (<Response [409]>)
I don’t quite understand the mechanics behind the sync, but I just run the command on the parent folder that has all the experiments. So, it essentially keeps syncing all my offline experiments.
My wandb version is wandb==0.15.4
I think a new flag max_retries should be introduced in wandb sync command where we can define the number of retries we want for a specific file sync. If I as a user anticipate that some runs can be missing, then I can just set it to 1. Currently, it tries around 7 times per failed attempt, which clogs the overall syncing process.
Others who face this issue: I have found a workaround. Essentially, I’m just tagging the runs as trash instead of actually deleting it, and then filtering them out in my UI. While this is not a clean solution, but it works for me.
@soumyasanyal Glad to hear tagging the runs provided a suitable workaround. I can make a feature request for setting max_retries, and I will open up this thread again if any progress is made on it.