the lines in my plots seem to be left-truncated after around 20k steps. Is this intended behavior or a problem on the logging side. I use the built-in logging of huggingface trainer and the model resumed after around 20k steps. But the resume starts at the most recent global step, so no values should be overwritten. When trying to export the graph data as csv, the cells are completely empty for those missing values (so not a 0.0, NaN, null or something).
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| Resumed run plotting broken | 3 | 167 | April 18, 2024 | |
| How to override existing statistics in plot when resuming from a checkpoint? | 3 | 628 | March 25, 2023 | |
| Tensorboard sync shows incorrect number of steps | 4 | 710 | April 4, 2023 | |
| All training curves are suddenly lost | 3 | 20 | July 19, 2024 | |
| Training plots disappeared after run is done | 8 | 873 | January 13, 2024 |