Synced runs still shows as unsynced

W&B Help
1

I have an offline run on my local machine.
If I run:
wandb sync --sync-all
it uploads the run to my wandb dashboard.

However, if I then run:
wandb sync --clean --clean-force --include-offline --include-offline --clean-old-hours 0
then it tells me that there is no run older than 0 hours.

If I run wandb sync --show 1, it still shows me the local run as unsynced.
I can run wandb sync --sync-all --include-offline --include-synced as many time as I want, the same behabior happens.

This is especially problematic because my main problem is that I want to append results on resumed runs, and this is already difficult to do with offline runs, which I need because my compute node doesn’t have internet connection, and long execution time are not allowed.
Is there any possibity to solve the syncing problem first then make the --append option be compatible with offline runs ?

I’m currently testing it with wandb 0.16.5

Thanks.

2

Hello streethagore,

Thanks so much for reaching out today.

When you run wandb sync --sync-all , it should upload all your runs to the W&B dashboard. However, if you immediately try to clean them with --clean-old-hours 0 , it might not find any runs older than 0 hours because they were just synced. The --clean-old-hours flag is intended to clean runs that were created a certain number of hours ago, not the ones that were just synced.

To ensure that your runs are indeed synced, you can check the W&B dashboard to see if the runs appear there after the wandb sync --sync-all command.

As for appending offline runs I will look into if we already have a feature request for that and if not I would be more than happy to submit one for you!

I hope this helps. Let me know if you are able to sort out the syncing issue or if you had any other questions.

Have a great rest of your week!

Best,
Jason

3

Thanks for your reply.
Unfortunately, even if I wait more than 24 hours, wandb sync --clean will not delete my offline runs that are synced.

For appending offline runs, there is an issue here: [CLI]: wandb sync with --append option fails to sync offline runs with the same id · Issue #5865 · wandb/wandb · GitHub
and here:
[Feature] Resume offline runs · Issue #2423 · wandb/wandb · GitHub