# Sync problems **URL:** \ **Category:** W&B Help\ **Tags:** wandb\ **Created:** [May 23, 2023, 6:54pm UTC](https://community.wandb.ai/t/sync-problems/4457 "2023-05-23T18:54:34Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![parisi](https://avatars.discourse-cdn.com/v4/letter/p/35a633/32.png) [@parisi](https://community.wandb.ai/u/parisi)\ **Post date:** [May 23, 2023, 6:54pm UTC](https://community.wandb.ai/t/sync-problems/4457/1 "2023-05-23T18:54:34Z")
I run wandb on a cluster. Job nodes don’t have internet so I need to run `wandb sync --sync-all` from login nodes. If I kill the job, the command keep syncing the killed runs over and over again: I run the command, the run is synced, I re-run the command (to sync new runs) and the old runs are synced again. I suspect it’s because how the job was killed, that maybe `wandb.finish()` was not called. Could that be it? I then try to selectively sync only runs belonging to a project with `wandb sync -p myproj` but it just says ```auto wandb: Number of runs to be synced: 46 wandb: Showing 5 runs to be synced: wandb: wandb/offline-run-20230523_123938 ... [list of the 5 runs] wandb: NOTE: use wandb sync --clean to delete 81 synced runs from local directory. wandb: NOTE: use wandb sync --sync-all to sync 46 unsynced runs from local directory. ``` But the 5 runs are not actually synced. I also try tried cleaning with `wandb sync --clean` but nothing happens, despite it says that there are 81 runs that can be deleted. ---
**Author:** ![luis\_bergua](https://avatars.discourse-cdn.com/v4/letter/l/dbc845/32.png) [@luis\_bergua](https://community.wandb.ai/u/luis_bergua)\ **Post date:** [May 26, 2023, 3:16pm UTC](https://community.wandb.ai/t/sync-problems/4457/2 "2023-05-26T15:16:22Z")
Hi @parisi, thanks for reporting this! I’ll try to reproduce this on my end to see exactly what’s going on here so, to make sure, the steps you’re following are: 1. Execute some offline runs (let’s say 10 for example) 2. Run `wandb sync --sync-all` 3. Kill that before all runs are synced 4. Re-run `wandb sync --sync-all` 5. Runs synced initially are synced again (so they are synced twice) In general, I would recommend you to both assign the run to a variable like `run = wandb.init()` and also call `run.finish()` at the end to ensure no processes are running on your background. ---
**Author:** ![parisi](https://avatars.discourse-cdn.com/v4/letter/p/35a633/32.png) [@parisi](https://community.wandb.ai/u/parisi)\ **Post date:** [May 26, 2023, 4:28pm UTC](https://community.wandb.ai/t/sync-problems/4457/3 "2023-05-26T16:28:31Z")
Thanks. I have `try / except` to call `finish()`, and it works if I kill my run with CTRL+C or if it crashes. However, if I close the terminal my run still appears ar “running” on wanbd UI. I guess this is also what happens if I kill SLURM jobs with `scancel`. ---
**Author:** ![luis\_bergua](https://avatars.discourse-cdn.com/v4/letter/l/dbc845/32.png) [@luis\_bergua](https://community.wandb.ai/u/luis_bergua)\ **Post date:** [June 2, 2023, 1:00pm UTC](https://community.wandb.ai/t/sync-problems/4457/4 "2023-06-02T13:00:32Z")
Thanks for sharing this @parisi. This is probably because when you close the terminal, `finish()`hasn’t been called yet so the run will keep running. Would it work for you to use `try / except` and kill the run with CTRL+C if needed? ---
**Author:** ![parisi](https://avatars.discourse-cdn.com/v4/letter/p/35a633/32.png) [@parisi](https://community.wandb.ai/u/parisi)\ **Post date:** [June 2, 2023, 10:14pm UTC](https://community.wandb.ai/t/sync-problems/4457/5 "2023-06-02T22:14:02Z")
My code already has try/except, and I think they work fine with CTRL+C but maybe not when I use SLURM scancel. I will investigate this more. Isn’t there a way to detect “dead” runs? Like passing a timeout argument to init such that if the run is not synced after X time it will be considered finished? EDIT: Yes, my code can catch CTRL+C without problems, but scancel maybe sends a different signal. ---
**Author:** ![luis\_bergua](https://avatars.discourse-cdn.com/v4/letter/l/dbc845/32.png) [@luis\_bergua](https://community.wandb.ai/u/luis_bergua)\ **Post date:** [June 7, 2023, 12:58pm UTC](https://community.wandb.ai/t/sync-problems/4457/6 "2023-06-07T12:58:22Z")
Hi @parisi, thanks for your answer! Currently this isn’t possible but I can create a feature request to add that argument to the wandb.init() function, would you mind giving me some details about your use case so I can share that with our Product Team? ---
**Author:** ![parisi](https://avatars.discourse-cdn.com/v4/letter/p/35a633/32.png) [@parisi](https://community.wandb.ai/u/parisi)\ **Post date:** [June 7, 2023, 7:59pm UTC](https://community.wandb.ai/t/sync-problems/4457/7 "2023-06-07T19:59:41Z")
Sure, thanks! Having this argument can fix potential problem if the machine where the code is running crashes and the `finish` command is never launched. If this doesn’t happen, the server sees the run as still “running”. Also, if one restarts the machine and does `wandb sync --sync-all` it keeps resynching the dead run. This is a waste of time. ---
**Author:** ![luis\_bergua](https://avatars.discourse-cdn.com/v4/letter/l/dbc845/32.png) [@luis\_bergua](https://community.wandb.ai/u/luis_bergua)\ **Post date:** [June 9, 2023, 12:05pm UTC](https://community.wandb.ai/t/sync-problems/4457/8 "2023-06-09T12:05:55Z")
Thanks @parisi! I just shared this feedback with our Product Team. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/1X/366b649231631dbab896843020da0056074ac79d.png) [@system](https://community.wandb.ai/u/system)\ **Post date:** [August 8, 2023, 12:06pm UTC](https://community.wandb.ai/t/sync-problems/4457/9 "2023-08-08T12:06:30Z")
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.