Hi,
I’ve been training a model for more than 2 days when all training and validation curves suddenly went missing. I noticed this after I created a copy of the validation results in my output directory (cp -r validation validation_1400), just to preserve the state of those results at the 1400th epoch. The model is still training, but all my charts have been reset as if I just started the training. What did I do wrong?
Best,
Istvan
Update: About an hour later, everything spontaneously returned to normal. This leaves me quite perplexed, as it wasn’t just a browser issue (Safari and Chrome both reported missing curves). Nevertheless, I’m glad that this got sorted.
Hello @inhuszar , thank you for reporting and for the update that it is working now. Let us know if you have further question, and will mark this as resolved. Feel free to reach us out again for any concern.
Hi @inhuszar , since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!