Hi,
I have played around with wandb a bit for torch training and I really like it!
There’s one thing I have not found a great solution for, though. I run cross validation and for each fold I want to log and display the train and val losses for each epoch so that I can compare runs. The best way to do this that I have found is to use a parent run and then a child run for each fold with group=parent.id. This works great and I can compare aggregated results between different runs or look at the individual fold traces for one parents.
Unfortunately this breaks down when I do sweeps. Apparently only one run can be active per process and my current setup interrupts the parent run before writing the summary results to it, which breaks the sweep.
How do others do it? I guess I have to make do with a single run for all folds? But how do you log the loss-over-epoch curves per fold so that you get a clean representation in the UI?
Any help would be appreciated. Thanks for reading this!
Best wishes,
Thomas