The same code runs successfully and in a normal speed before, but last week, I have met the problem like: wandb: 429 encountered (Filestream rate limit exceeded, retrying in 2.441980818670473 seconds), retrying request. After that, it runs extremely slower than before. Some runs does not report this issue in the logs runs much slower too(running the same code) .
I tried to use ‘wandb init’ , start a new project, kill the processes and only keep <10 runs running… but it does not solve this problem.
Meanwhile, some runs take>3days to stop. It is stopping currently.
Could you help me, please? Thank you very much~!