Hello, I would like to discuss a problem in which newly created runs take a very long time to appear in the Workspace.
The individual runs themselves exist on the Web, so when I access the URL of a run directly from the CLI, I can see the data in my browser. However, there is a very long lag time (at least 30 minutes~) before the run is reflected in the Workspace.
I have not applied any filters on the Workspace, and the usage is only about 1GB out of 100GB. I also tried changing browsers, but that didn’t solve the problem.
Yesterday, I was examining the behavior of early_terminate in WandB Sweep, so I thought it was possible that it was caught in some kind of limitation. However, I did not run WandB Sweep in parallel on multiple computers, but only on one computer. Moreover, the data of the run itself seemed to be instantly recorded on the Web, so I thought that the possibility of a limitation was low.
The only problem is that the data is reflected too slowly in the Workspace.
We would appreciate it if you could investigate this issue.
I also want to add, that if one deletes a run, it gets deleted in the backend but the deleted runs still show up in the project run list. If you click on the deleted run you get redirected to a 404 page.
Got the following result. You see that the name of the run in the path on the left upper corner is also wrong. The actual name of the run was “rich-sponge-23”. Can’t include a second image since I’m new in the forum
Same problem here. I thought it was something related to the number of runs in the project or workspace or maybe even the server internet itself but tried different servers and making new projects and it has the same issue: runs take a long time to appear and even if they appear they seem to be bugged out as they do not show certain output metrics. Also can’t seem to move runs across projects or even delete them.
I hope you’re doing well. I wanted to reach out and provide an update regarding the recent performance issues we experienced. Over the weekend, we underwent some maintenance and migrations, which may have impacted our service.
Could you please let me know how things are on your end now? Are you still encountering any performance difficulties, or has the situation improved? Your feedback is invaluable to us.
Thank you for your response. I am relieved to know that the issue was only temporary. Everything is functioning normally now. I sincerely appreciate your help.
