Hi, I am currently recording a bunch of observations during a run via running over multiple seeds then using wandb.log.
I have some conditional observations I want to plot, where I have an indicator variable for when the event X occurs in the run. I want to see how another statistic changes when that X vs. when it isn’t, and ideally also how correlated that statistic changes/is correlated with X vs. how it changes/is correlated with a second indicator Y during an online training run.
Is there an easy way to do this? I logged the statistics, but X and Y are not available to be used as ‘group-keys’ which was my first thought to tackle the problem.
I looked into both the custom chart and weave. I couldn’t easily figure out the api for weave, but the custom chart seemed highly promising, except for the fact that I couldn’t group these variables, and I couldn’t extract prediction intervals over multiple runs.
The other issue is ‘X’ vs. ‘Y’ only occur at a small subset of times and most of the time, therefore, if you run say np.corrcoef, you get a bunch of NANs due to no variance.