All records are lost in a project without any action

Dear Sir or Madam,

Sorry for bothering you, I think there is an error in one of my wandb projects and the records of all runs were lost. The account is nbower0707, email 1155156871@link.cuhk.edu.hk, and the project name is ocp22.

Everything worked fine before today, and I did a lot of experiments on this project. I’m uploading records of my metric around every 5000 steps, and the result validation metric plot should be something like figure 1 shows(continuous lines of records, with multiple data points) I’m uploading the corresponding metrics every 2500 steps, and wandb displayed all results fine yesterday (either undergoing or finished runs)

However, when I check the plot today, the record of metric in all runs were (completely or partly) lost, except for some small isolated data points left (as figure 2 and 3 shows).

I tried to use wandb sync from the local file, and upload the runs to a new project, the result is still the same.

I didn’t do any specific operations regarding wandb logging process or on the website. The project consist of runs uploaded from different machines, therefore it wouldn’t be mistakenly deletion/ false operation offline. And the phenomenon of lost of data also occurs on old runs that finished weeks ago.

Please let me know if you have any suggestions on this error, and if the records could be recovered.

Your time and patience are sincerely appreciated.

Bowen Wang

I have the same problem…

Me too. From night to morning the runs graphs miss a lot of data points in the validation section and I also noted that the resize of the panels in that section doesn’t work properly

Same problem here, the plots are weird and loses a lot of data points

Me too… It seem to be same problem

Hi Everyone,

Apologies for the inconvenience here. We are looking into the issue - any links to workspaces where you see this currently would be greatly appreciated.

Thanks,
Ramit
W&B Support

2 Likes

Mine is Weights & Biases

Same problem here, plots look similar to the ones shown

Here’s mine

The same issue…
Yesterday they were fine…

Same issue here. Though the charts look fine when opened in their respective run, just not in the combined workspace…

Now it’s fine for me, thank you for the support

Hey all,

Our engineering team looked into this and rolled back some changes, everything should be working fine now.

Please let us know if this issue persists.

Thanks,
Ramit

3 Likes