Hi! I’m currently running around 400 jobs simultaneously, and each one tries to log to WandB. I attempted to write custom asynchronous logging, but I’m still facing the same error when logging in online mode. This issue causes all runs to be 2-3 times slower than they would be without logging.
Is there any common workaround that would allow syncing information online without hitting the Filestream limit?
Hi @kotekjedi
Good day and thank you for reaching out to us. Happy to help you on this!
Rate limits are preventive measure that we apply in order to maintain site stability. What I can suggest from here is try to reduce the number of jobs being executed so you can avoid being rate limited.
You may want to check this guide to get more details on how limit works in Weights and Biases.
For the meantime, you can try to run your experiments on a new project so you can proceed. Rate limits are being applied on the project level. I hope this helps.
Thanks,
Paulo
Hi @kotekjedi , since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!