Hey everyone,
Let’s say that I have a dataset with 50000 samples and I am training my model for 10 epochs. Now, in each epoch, I am recording the per sample loss (i.e. loss of each sample - Not the average loss of all samples). This means that there are 50000 loss values per epoch. I want to log these values for each epoch, so that I can later perform some analysis on how the loss values for the samples change as training progresses (And, if possible, observe the loss values of a particular sample across epochs). For reference, this paper tracks such statistics. Here are two ways I can think of doing this -
A simple way to do this is to update the values in a 50000x10 array, then log the array as a table at the end of training (I would obviously need to track which indices belong to which samples). However, I need to wait for the training to end in this scenario.
I can also log each sample’s statistic with wandb.log (Maybe put them under “sample_statistics/” to pull them more easily). This ensures that the metrics are logged as and when they are observed, however, I am not sure if this is the most optimal solution.
Is there any other way in which I can do this so that I can analyse the resulting data effectively? Open to all suggestions!
Thank you!
Thank you for writing in with your question. Ideally what would be best in your case here is to create an Empty table and log per sample loss values per epoch and be able to see your data live in the UI. However, we currently, don’t support adding new rows to existing tables that you’ve already logged. We are working on adding this functionality.
In the meantime here are two approaches
Keep the wandb.Table locally holding all the data in memory and logging it once.
Keep logging the same table at each step, and just add new rows to it. The final table you log will have all the rows you want, and you’ll be able to see the latest table logged in the UI. This would be risky if you have large table sizes.
Please Note: If you were to look through our docs and come across the Add Data Incrementally to Tables doc, this functionality is currently broken and we are working on an active fix. There is github issues thread here where community members have posted workarounds for this, you may find it helpful.
Hi @tataganesh, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!