Hello, I just encountered this problem:
429 encountered (Filestream rate limit exceeded, retrying in 9.7 seconds.), retrying request
And it’s very strange that when I open the wandb, the loss becomes nan.
If I comment out this line of code, everything becomes normal：
loss = criterion(outputs, labels)
# wandb.log({"loss":loss.item()})
Hi @rainy-autumn-fall ,
Thank you for reporting. Our engineers were aware of this and took care of it. May I confirm if it is now working on your end?
Regards,
Joana Marie
Hi @joana-marie
Thanks for your reply. Everything’s normal now. But I still hope your engineers can keep an eye on this issue, there will be a few hours of not being able to use wandb when it comes up.
Regards,
Rainy