Rate Limit Exceeded

Hello,

Since recently, I have been getting this error when browsing the runs/workspace of my latest project.

For info, it is a segmentation tasks trained in pytorch using fastai callback.

My log looks like this:


0: *Quadro RTX 8000,         48.6GB, tensor_cores=72
1: GeForce RTX 2070 SUPER,   8.0GB, tensor_cores=40
 Selecting GPU : Quadro RTX 8000
Exception ignored in: <function _releaseLock at 0x7f3232131ee0>
Traceback (most recent call last):
  File "/home/tcapelle/miniconda3/envs/pytorch_18/lib/python3.8/logging/__init__.py", line 227, in _releaseLock
    def _releaseLock():
KeyboardInterrupt:
WandbCallback requires use of "SaveModelCallback" to log best model
wandb: 429 encountered ({"error":"rate limit exceeded"}), retrying request
wandb: Network error resolved after 0:00:08.499452, resuming normal operation.
wandb: 429 encountered ({"error":"rate limit exceeded"}), retrying request
wandb: 429 encountered ({"error":"rate limit exceeded"}), retrying request
wandb: Network error resolved after 0:00:09.337473, resuming normal operation.
wandb: 429 encountered ({"error":"rate limit exceeded"}), retrying request
wandb: Network error resolved after 0:00:08.679539, resuming normal operation.

this happens in the training notebooks and in the wandb page afterwards.

1 Like

Thanks for flagging! We’re looking into it :tea:

1 Like

It appears the issue is due to logging either too frequently from one run, from too many runs in parallel, or loading too much data in the UI. If we can track down which one it is, that will help debug. -Is it possible for you to share the run page/link? That might help debug the same.

The team had also deployed a fix related to the same today, could you please confirm if you’re facing the issue today as well?

TIA! :slight_smile:

Thanks, Thomas. Since the link is visible to everyone, I deleted your comment to keep it private. I hope that’s okay.

I’ll get back once the team takes a look :slight_smile:

1 Like

It appears to be working fine now.

1 Like

Thank you, please let me know incase this/any other issues reappear :smiley: