# All training curves are suddenly lost

**URL:** <https://community.wandb.ai/t/all-training-curves-are-suddenly-lost/7263>\
**Category:** W&B Help\
**Tags:** wandb\
**Created:** [July 15, 2024, 5:56pm UTC](https://community.wandb.ai/t/all-training-curves-are-suddenly-lost/7263 "2024-07-15T17:56:27Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![inhuszar](https://avatars.discourse-cdn.com/v4/letter/i/ac8455/32.png) [@inhuszar](https://community.wandb.ai/u/inhuszar)\
**Post date:** [July 15, 2024, 5:56pm UTC](https://community.wandb.ai/t/all-training-curves-are-suddenly-lost/7263/1 "2024-07-15T17:56:27Z")

</div>

Hi,

I’ve been training a model for more than 2 days when all training and validation curves suddenly went missing. I noticed this after I created a copy of the validation results in my output directory (cp -r validation validation\_1400), just to preserve the state of those results at the 1400th epoch. The model is still training, but all my charts have been reset as if I just started the training. What did I do wrong?

Best,  
Istvan

---

<div class="post-metadata">

**Author:** ![inhuszar](https://avatars.discourse-cdn.com/v4/letter/i/ac8455/32.png) [@inhuszar](https://community.wandb.ai/u/inhuszar)\
**Post date:** [July 15, 2024, 6:09pm UTC](https://community.wandb.ai/t/all-training-curves-are-suddenly-lost/7263/2 "2024-07-15T18:09:36Z")

</div>

Update: About an hour later, everything spontaneously returned to normal. This leaves me quite perplexed, as it wasn’t just a browser issue (Safari and Chrome both reported missing curves). Nevertheless, I’m glad that this got sorted.

---

<div class="post-metadata">

**Author:** ![joana-marie](https://avatars.discourse-cdn.com/v4/letter/j/f6c823/32.png) [@joana-marie](https://community.wandb.ai/u/joana-marie)\
**Post date:** [July 17, 2024, 6:22am UTC](https://community.wandb.ai/t/all-training-curves-are-suddenly-lost/7263/3 "2024-07-17T06:22:05Z")

</div>

Hello @inhuszar , thank you for reporting and for the update that it is working now. Let us know if you have further question, and will mark this as resolved. Feel free to reach us out again for any concern.

---

<div class="post-metadata">

**Author:** ![joana-marie](https://avatars.discourse-cdn.com/v4/letter/j/f6c823/32.png) [@joana-marie](https://community.wandb.ai/u/joana-marie)\
**Post date:** [July 19, 2024, 9:29am UTC](https://community.wandb.ai/t/all-training-curves-are-suddenly-lost/7263/4 "2024-07-19T09:29:33Z")

</div>

Hi @inhuszar , since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!