I have many runs saved locally without W&B. Each of them is in a different folder with a cfg.yaml file (hyperparameters configuration) and many npz files (with different statistics arrays, such as “loss” “regularizer loss”, …).
What’s the fastest way to upload them to W&B to analyze them with time plots?
Currently, I loop through the folders, read the configuration, and upload npz files one by one. For each file, I have to loop through the timesteps of the run. That is, I have to loop through the values of the arrays and call wandb.log. This is EXTREMELY slow (I have 10000 runs and it takes 100 hours).
Alternatively, I can make a pandas dataframe where entries are arrays and upload to W&B as a Table, but then I don’t see how I can make time plots out of them.