Dear Sir or Madam, Sorry for bothering you, I think there is an error in one of my wandb projects and the records of all runs were lost. The account is nbower0707, email 1155156871@link.cuhk.edu.hk, and the project name is ocp22. Everything worked fine before today, and I did a lot of experiments on this project. I’m uploading records of my metric around every 5000 steps, and the result validation metric plot should be something like figure 1 shows(continuous lines of records, with multiple data points) I’m uploading the corresponding metrics every 2500 steps, and wandb displayed all results fine yesterday (either undergoing or finished runs) However, when I check the plot today, the record of metric in all runs were (completely or partly) lost, except for some small isolated data points left (as figure 2 and 3 shows). [Picture 1] I tried to use wandb sync from the local file, and upload the runs to a new project, the result is still the same. I didn’t do any specific operations regarding wandb logging process or on the website. The project consist of runs uploaded from different machines, therefore it wouldn’t be mistakenly deletion/ false operation offline. And the phenomenon of lost of data also occurs on old runs that finished weeks ago. Please let me know if you have any suggestions on this error, and if the records could be recovered. Your time and patience are sincerely appreciated. Bowen Wang

Hey all, Our engineering team looked into this and rolled back some changes, everything should be working fine now. Please let us know if this issue persists. Thanks, Ramit

Me too. From night to morning the runs graphs miss a lot of data points in the validation section and I also noted that the resize of the panels in that section doesn’t work properly

Same problem here, the plots are weird and loses a lot of data points

Me too… It seem to be same problem

Hi Everyone, Apologies for the inconvenience here. We are looking into the issue - any links to workspaces where you see this currently would be greatly appreciated. Thanks, Ramit W&B Support

Same problem here, plots look similar to the ones shown

The same issue… Yesterday they were fine…

Same issue here. Though the charts look fine when opened in their respective run, just not in the combined workspace…

All records are lost in a project without any action

W&B Help

Topic		Replies	Views
Training curves all lost suddenly W&B Help dashboard , wandb	4	213	September 26, 2024
Wandb stops uploading data W&B Help wandb	19	2324	February 29, 2024
Lost a lot of metric data points in the past few hours W&B Help dashboard , wandb	4	1103	July 5, 2023
Training plots disappeared after run is done W&B Help dashboard , projects , wandb	7	1070	November 14, 2023
Records partly lost in training runs W&B Help	3	430	March 5, 2024

All records are lost in a project without any action

Related topics