Hey there 
During my training I want to produce some visual outputs every epoch to monitor that the predictions make sense. I am logging a matplotlib figure with:
logger.log({'last epoch': fig})
The issue is that wandb is saving every figure that is produced instead of overwriting the last one. How can this be achieved?
Thanks in advance
Arturo
Hi, @artuntun , right now you are logging the figure to the key called “last epoch”. If you just log it normally like wandb.log({'epoch': epoch, 'predictions_fig': fig}) then you will be able to see the most recent figure but also have the option to toggle back through all the other figures at each epoch to see how your predictions evolved overtime.
that is the behavior that is already happening. What I actually want is to not save all of them. It a heavy figure and I don’t want to save that much data.
Ahh I see. In that case, I would recommend only logging your final image. By design, it is difficult to delete or overwrite data logged to a run, during a run. This is to prevent accidental data loss and preserve as much of the experiment as possible. Alternatively, if you just wanted to log the image file, you could save it with wandb.save(path-to-image).
thanks for the response aidanjd
Is it possible to log the final image if I stop the training loop manually?
Hi @artuntun , saving the image with wandb.save() will achieve this naturally though your image won’t appear in the workspace panels. The other way of doing this would be to catch whatever interrupt errors you are throwing to stop you training loop and call your end of training image logging then.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.