Wandb Errors when I try to execute mlflow command

W&B Help
1

Command I run and errors I get below

mlflow run . -P steps=download

wandb: While tearing down the service manager. The following error has occurred: [Errno 32] Broken pipe

wandb: ERROR Abnormal program exit

wandb: ERROR Internal wandb error: file data was not synced

I get several errors like the following as well but won’t post all right now.

  File "/home/adduser/miniconda3/envs/mlflow-c1beca2aef7a523d139ba0209b68828b5e298901/lib/python3.10/site-packages/wandb/sdk/internal/internal_util.py", line 49, in run
    self._run()

Any insight into why mlflow isn’t communicating with wandb even after being logged in would be helpful.

2

Hi @samueltwjohnson Good day and thank you for reaching out to us. Happy to help you on this!

May I know what is the host URL you are using for your instance? Can you share what you have set for WANDB_BASE_URL?

Thanks,
Paulo

3

Hi Paulo,

I think I got what you are looking for. Please see below.

https://api.wandb.ai

If this is not what you are looking for, I apologize. Mlflow and Wandb are new to me.

Thank you,
Samuel

4

Hi @samueltwjohnson Good day and thanks for your update! Would you mind changing that URL into https://app.wandb.ai and let me know if this fixes the issue?

5

Hi Paulo,

Unfortunately, I get the below error.

wandb: ERROR https://app.wandb.ai is not a valid server address, did you mean https://api.wandb.ai?

Thank you,
Samuel

6

Hi @samueltwjohnson Thank you for checking this for me, you may now revert the changes back using the original setting. We will need to further investigate this and I may need to request some more information that might help us. Could you please provide us the following?

  • Full stack trace of the initial errors you are getting
  • debug-internal.log and debug.log for the affected run. These files are under your local folder wandb/run-_-/logs in the same directory where you’re running your code. These files will help us with more details about this error.
  • A code snippet on how you are trying to execute your MLflow command

Thank you!

7

Hi @samueltwjohnson , since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!