Hi everyone,
I have been using W&B to log my outputs of experiments and in one case I have a set up with fixed code and the hyperparameters change and the same code is run on the same data for the same amount of steps. However, when I use my own function to retrieve the values logged during training, they are of different lengths and it’s causing quite a headache to plot these metrics over time because they don’t align naturally.
I use my own function to retrieve a metric from a run:
def get_wandb_history(identifier, key):
run = api.run(identifier)
run_history = run.history()
key_history = run_history[key]
key_history = np.array([x for x in key_history if float(x) > 0.0])
return key_history
The penultimate line is because of the NaNs that store info at steps in between epochs.
I ran 6 different runs where only a single hyperparameter changed. After retrieving the results via the function above, the lengths I have for them are:
61
62
64
67
60
61
What’s going on here? Is there a better / more reliable way to check the stored outputs? In the visualisation tool online, the runs are aligned perfectly and show the effect I was hoping to find. However, when I want to use those values in my own plots, they are not temporally aligned. So, they are being stored correctly for visualisation on the web interface - just not when I want to retrieve them (?)
Thanks.