Rate limit exceeded -- wandb

W&B Help
1

Hi,

Since yesterday, I’ve been getting this error when trying to access my dashboard:

Screenshot 2023-12-18 at 11.46.42 AM
Screenshot 2023-12-18 at 11.46.42 AM1516×466 19.2 KB

My collaborator is also getting a similar error message & inability to access the dashboard.

Any tips to get this working again?

2

Hi @milani ,

Thank you for reaching out for support. I’ll be happy to assist you with this. Can you provide a list of your collaborators please and we will investigate this further.

Regards,
Carlo Argel

3

Hi Carlo,

Thanks for helping! My collaborator is gabr1e1 and the team name is interpretablemarl.

4

Hi @milani, thank you for the additional input. The reason you were seeing this is that our backend team had to disable your team while we investigated why some of your runs were causing instability on our site. It appears that some of your runs were logging entire data sets as a large array for a key called dataset , which violated our limits as per Experiments Limits and Performance. We are working on changes to our system to prevent this instability from happening again when logging these arrays. We’ve enabled the team and it’s users, but for now we recommend that you consider using media files or artifacts to log your datasets; otherwise, we might have to disable the team again if runs disrupt the site.

Please let me know if you have any questions and we appreciate your understanding on the matter.

5

Thank you for explaining the issue! I’ll make sure that we adjust the code accordingly.

Quick question: if we have completed some offline runs with the dataset key, how can we remove them from the log before syncing the runs?

6

@milani , I will get back to you on this, will look into the following:

  1. If the sdk team has a method of modifying offline runs to remove those key:values
  2. If the backend team needs you to wait to sync those runs.

Will have an answer by EOD

7

Thank you! We will refrain from syncing those runs until we have a response. We really don’t want to cause any more issues with your system.

8

Hi @milani , thank you for understanding and appreciate your response. At this time you cannot sync the offline runs until our updates are made in the new year to our backend. As for updating the configs of offline runs, as our sdk team is out during the holiday break, soonest I can get you an answer is the new year.

Happy holidays!