This morning we were looking at the visualizations and charts on an active training run and everything was fine. After about 11am PDT, all of the visualizations started randomly disappearing. Sometimes only the loss charts would be visible, other times the losses and metrics and statistics would all be visible.
The best I can tell is that the site is only showing charts for whatever things were in the most recent step. If you send some things less frequently, then their charts/visualizations disappear until they’re in the step data again.
Exactly the same issue is happening on my side. Roughly about 11 am (PDT) some of the training and validation visualizations started to disappear for finished runs. In some cases only the last epoch metrics were visible but in some cases only one metric was visible.
Hi @cogwheel and @mehraveh,
Thank you for reporting and sorry for the issue. This is a current issue with our backend that is being fixed now. I’ll update here shortly once this has been resolved.
Thank you,
Nate
@nathank has this been fixed globally? I seem to be running into a similar issue.
I log a metric after every epoch in the validation loop (and these 25hour epochs). I see the metric I logged in my summary section on the info page, but I don’t see the chart with that metric on the charts page. And, I can’t seem re-add it.
@fishbotics the database is currently backfilling any metric information that was uploaded during the issue. Our engineering team said this may take a few hours for this process but your data should show up in the UI shortly. Feel free to ping me here if you notice after a couple of hours that you still are having issues.
Thank you all for reporting this so we could get this resolved as quickly as possible
@cogwheel Could you possibly send me a link to your workspace? If you don’t want to share here you can also email support@wandb.com and explain the issue and I can respond via email.