CommError: Error uploading run: returned error 40

wandb.init fails with

CommError: Error uploading run: returned error 403: {"data":null,"errors":[{"message":"\u003c!doctype html\u003e\u003cmeta charset=\"utf-8\"\u003e\u003cmeta name=viewport content=\"width=device-width, initial-scale=1\"\u003e\u003ctitle\u003e403\u003c/title\u003e403 Forbidden"}]}

wandb==0.21.0
api key is provided via wandb login

A few hours ago it worked, seems like some external error:(

Hi there! I encountered the same issue. For context, I was training with various GPUs and various models (24).

What I deduce from the amount of time I spent resolving this issue is that using a different API key (from a different wandb account), I was able to run 4 runs simultaneously. However, on my 5th run, it did not work. However, after changing the wandb accounts, it worked. Or maybe because I used a different GPU (a 4080super rather than a 4090). However, I still believe that it might be caused by simultaneous runs under one account, even if Wandb does not impose a hard limit on the number of runs that can be run concurrently with one API key.

I hope this helps people who have experienced the same issue. My solution for sure isn’t the best, but it worked for me, but sadly wasted a lot of time and money on my rented GPUs lol