I run into this problem today. It seems the ‘fix’ is a manual rate increase on the account? Could I get a rate increase? It’s breaking my training.
More importantly, I am not sure why this is triggered and how to fix it on the user side. My code use wandb.log() only at the end of an epoch, which is about once every 2.5 hours, how it could exceed the rate limit?
Also, the error is triggered while the epoch is still running, so something triggered the error when wandb.log() havent been called?
New to wandb, what am I missing?