I am running some experiments where I have multiple random seeds per experiment setting, so I am trying to group the runs together to get their average and standard deviation (this is standard in reinforcement learning research these days). However, I can’t seem to figure out how to get this to reliably work on wandb – sometimes it works and sometimes it doesn’t.
For reference, this is the kind of plot that I am trying to generate:
There are two overall curves, but these are averaged among several runs, which is why you see a shaded region for standard deviation.
I make this figure in a wandb report by going to the panel grid and assigning different runs together to a group manually. Here is a screen recording of the process of how I try to do this.
Here, I’m showing another set of runs that I’m trying to group together (it’s about 15 total individual runs, but in 3-4 groups, so I’m trying to group the curves). However, clicking on the “Runs” button means nothing changes! This is strange since it’s how I do this to create my other grouping plots. Sometimes it works, sometimes it does not. Does anyone have suggestions on how to make this function work more reliably? Thanks!
To grouping using wandb, a user will have to perform any of the following:
sett a group id in script for the runs
set a group environmental variable within your project that is shared by all runs in the group
The above approaches will automatically group runs in the UI when logged.
When grouping through the UI for groups that were not initially set to share a group via script, you first group by a common metric, then within the chart that now shares the runs, aggregate across them.
Hi! Thanks so much for the advice.
To be specific here is the report that I used which I showed in the screen recording video:
This report links to a bunch of runs in the workspace that I am using.
I updated the report today (July 15, 2022, or 07/15/2022) with a link to this post, to indicate where I was trying to group runs.
To clarify, these are for runs that (I believe) were not initially set to share a group via a script. I think this is the case you are talking about with “grouping through the UI for groups that were not initially set to share a group”.
I think the relevant part for me is “Grouping dynamically in the UI”. However the challenge in my case is that I am not trying to group by a hyperparameter. I’m trying to group an arbitrary set of runs that may or may not share some hyperparameters.
Also is there a way to do this in a wandb report after it has been created? I can actually somewhat reliably (I think) group runs if I accumulate the set of runs I want (by clicking on the “eye” button to the left of each run in the table) and make a new wandb report. However sometimes I want to modify existing reports.
Thank you for providing clarification @danieltakeshi and providing the report link. I do recognize that there is a sight limitation in our platform in allowing users to group runs after they had completed them when they don’t have any obvious distinguishing features the runs could be grouped against. We currently have a feature request in place to expand grouping functionality post logging to include arbitrary grouping. I will add you to this ticket and provide an update when available.
Thanks for clarifying, and it would be great if such a feature is available.
I am able to group reliably if I click on ALL the runs I want to consider for any possible groups, then I make a report from the existing stats that are logged. If I am able to do that then it seems like clicking dynamically on the wandb report UI will group them correctly. Also, interestingly, when I do this I am also able to add future runs.
So the key seems to be: make an initial report with the statistics I want to log already there and an initial set of runs to group in whatever way. Then I can group and add more runs later. If I try to add other statistics later, it will not let me group. It’s a lot of trial and error but I think I have something reliable, happy to clarify if needed.
