Running Sweep stops running new runs every 5 runs and each sweep creates 5 agents and now only 1

W&B Help
1

I was using WandB without any issues until yesterday. After upgrading my account to exceed 100 GB, my sweeps containing 30-100 runs have started stopping unexpectedly. Despite no crashes or failed runs, each agent now only completes 5 runs, whereas previously, each agent would complete 93-99 runs. (the sweep just stops running new runs until I run the sweep again with nohup wandb agent xxxxx &) Running “nohup wandb agent xxxxx &” every 5 runs (now only 1 run) which was previously around 99 runs)

Screenshot 2024-08-03 at 14.54.52
Screenshot 2024-08-03 at 14.54.523314×764 88.1 KB

There is no max count limit in the sweep it is a grid method
The premature stopping of experiments is significantly impacting my research. Thank you in advance for your assistance.

2

Hey, thanks for reaching out!

Could you provide the following information:

  • which wandb version are you using
  • are you using wandb core?
  • could you share the code you are using for your sweep?
3

Hi, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!