Dear Sir or Madam,
Sorry for bothering you, I think there is an error in one of my wandb projects and the records of all runs were lost. The account is nbower0707, email 1155156871@link.cuhk.edu.hk, and the project name is ocp22.
Everything worked fine before today, and I did a lot of experiments on this project. I’m uploading records of my metric around every 5000 steps, and the result validation metric plot should be something like figure 1 shows(continuous lines of records, with multiple data points) I’m uploading the corresponding metrics every 2500 steps, and wandb displayed all results fine yesterday (either undergoing or finished runs)
However, when I check the plot today, the record of metric in all runs were (completely or partly) lost, except for some small isolated data points left (as figure 2 and 3 shows).
I tried to use wandb sync from the local file, and upload the runs to a new project, the result is still the same.
I didn’t do any specific operations regarding wandb logging process or on the website. The project consist of runs uploaded from different machines, therefore it wouldn’t be mistakenly deletion/ false operation offline. And the phenomenon of lost of data also occurs on old runs that finished weeks ago.
Please let me know if you have any suggestions on this error, and if the records could be recovered.
Your time and patience are sincerely appreciated.
Bowen Wang