the lines in my plots seem to be left-truncated after around 20k steps. Is this intended behavior or a problem on the logging side. I use the built-in logging of huggingface trainer and the model resumed after around 20k steps. But the resume starts at the most recent global step, so no values should be overwritten. When trying to export the graph data as csv, the cells are completely empty for those missing values (so not a 0.0, NaN, null or something).
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| Training Plots Disappear once the run is done | 5 | 329 | December 9, 2024 | |
| Resumed run plotting broken | 3 | 170 | April 18, 2024 | |
| How to override existing statistics in plot when resuming from a checkpoint? | 3 | 656 | March 25, 2023 | |
| Tensorboard sync shows incorrect number of steps | 4 | 714 | April 4, 2023 | |
| All training curves are suddenly lost | 3 | 21 | July 19, 2024 |