Hello,
I have the following situation: I performed a bayesian sweep where I didn’t really know how much runs the hyperparameter optimization should take, but estimated (based on a past project) that it shouldn’t take more than a 100 runs, so this is what I set the run_cap to. I then realised that after I performed 100 runs, the values (of my metric, but especially of the hyper parameters tried) did not converge quite yet (probably because I ran too many of them in parallel), so now, I would like to continue the sweep with more runs for the values to converge. However, I cannot start new runs for the sweep now because it is marked as finished – because of the run cap. Even explicitly resuming it (via “Resume” in the interface) does not help: While trying a new run now doesn’t crash anymore with the error message that the sweep has finished, the sweep still does not send new jobs, so the program with the agent just terminates without performing a run.
So my question would be: Is it possible to ad hoc change the run_cap of a finished (or running) sweep, and if so, how would I go about doing that?
(If it’s not possible, I guess starting a new sweep from the existing runs in the previous one should be possible as well?)
Thanks in advance!
Daniëlle
Hello, This a reply from our support bot designed to assist you with your Weights & Biases related queries. To reach a human please reply to this message.
To address your situation with the Bayesian sweep and the need to continue beyond the initially set run_cap, here are the steps and considerations for managing and extending your sweep in Weights & Biases:
-
Increasing the run_cap of a Sweep: Currently, the Weights & Biases documentation and available resources do not explicitly mention a direct method to increase the
run_capof a sweep that has already reached its limit and is marked as finished. This limitation suggests that modifying therun_capof a sweep after it has been initiated might not be directly supported through the Weights & Biases interface or API. -
Resuming a Finished Sweep: While you’ve noted that using the “Resume” option in the interface does not initiate new runs as expected, this aligns with the understanding that once a sweep is marked as finished (due to reaching its
run_cap), it may not be designed to dynamically accept new runs beyond this cap without specific support for such an operation. -
Starting a New Sweep from Existing Runs: Given the constraints mentioned above, a practical approach to continue exploring the hyperparameter space with additional runs would be to start a new sweep, potentially using the configurations and insights gained from the current sweep. While this is not the same as extending the
run_cap, it allows for continuity in experimentation. You can reference the configurations of your current sweep to set up the new one, adjusting parameters or therun_capas needed based on your revised expectations. -
Best Practices for Managing Sweep Run Limits:
- Estimation and Adjustment: Initially, estimate the number of runs conservatively, considering the complexity of the model and the hyperparameter space. If unsure, it’s better to set a higher
run_capto avoid prematurely ending the sweep. - Parallel Runs Consideration: Running many sweeps in parallel can affect the convergence and performance of each run. Adjusting the degree of parallelism based on available resources and the nature of the sweep (e.g., Bayesian optimization) might provide more consistent results.
- Monitoring and Analysis: Regularly monitor the progress and results of your sweep. Early analysis might indicate whether the sweep is converging as expected and if adjustments are needed.
- Estimation and Adjustment: Initially, estimate the number of runs conservatively, considering the complexity of the model and the hyperparameter space. If unsure, it’s better to set a higher
Unfortunately, without direct support for increasing the run_cap of an existing sweep, starting a new sweep with adjusted parameters or a higher run limit appears to be the most straightforward way to continue your exploration. For detailed guidance and potential updates on features, consulting the Weights & Biases documentation and reaching out to the Weights & Biases support or community forums might provide additional insights and assistance.
To reach a human please reply to this message.
-WandBot 
Hi WandBot,
Thank you for your answer!
I couldn’t find anything in the documentation either, that’s why I asked. However, I’m not sure this doesn’t mean that there isn’t an option that’s just not that well documented yet. Thus, it would be great if a human could take a look at my question as well.
Hi @danielle-schuman Good day and thank you for reaching out to us. Happy to help you on this!
Please note that sweep configs cannot be modified after being created. What you could do:
- Create a new sweep from the UI (the Run Table has that option) with the same config (changing the run cap)
- Use the
Add prior runsoption to add all the Runs from the finished Sweep onto the new one (they should be able to filter them by sweep to find them).
Let us know if these can help!
Hi @paulo-sabile, Thank you, that’s good to know!
I have done as you suggested and I think that it’s working fine.
Good day to you, too!
Thank you @danielle-schuman! I am happy to hear that its working for you. I will now mark this as Solved. Please feel free to reach out to us again if you need assistance.
Have a great weekend!
Best Regards,
Paulo