I want to find the best hyperparameters using wandb, however, some combinations of them raises cuda memory error, how could I tell wandb that still runs with new hyperparameter combinations if there are these errors? So I do not need to check that all possible combinations do not raise a memory cuda error. I am afraid that the whole sweep will stop after a specific number of runs has failed.
Could I use some try except or tell wandb to always execute new runs (like “allow unlimited failed runs”) ?
Thanks in advance