Please see below code. When trying to run a download step, I get several wandb errors. I am on VS Code using WSL Ubuntu 20.04.
mlflow run . -P steps=download
2024/07/17 06:12:19 INFO mlflow.utils.conda: Conda environment mlflow-f6548607417935a06d9b14a98cd4db9af6ea5e32 already exists.
2024/07/17 06:12:19 INFO mlflow.projects.utils: === Created directory /tmp/tmpqp998j3m for downloading remote URIs passed to arguments of type 'path' ===
2024/07/17 06:12:19 INFO mlflow.projects.backend.local: === Running command 'source /home/adduser/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-f6548607417935a06d9b14a98cd4db9af6ea5e32 1>&2 && python main.py main.steps=\'download\' $(echo '')' in run with ID '5f9a379efe9345bd9459b4c4d4a3e9dc' ===
2024/07/17 06:12:21 INFO mlflow.projects.utils: === Fetching project from https://github.com/udacity/Project-Build-an-ML-Pipeline-Starter.git#components/get_data into /tmp/tmpfxs0kz77 ===
2024/07/17 06:12:22 INFO mlflow.projects.utils: Fetched 'main' branch
2024/07/17 06:12:23 INFO mlflow.utils.conda: Conda environment mlflow-c1beca2aef7a523d139ba0209b68828b5e298901 already exists.
2024/07/17 06:12:23 INFO mlflow.projects.utils: === Created directory /tmp/tmpranvfhsx for downloading remote URIs passed to arguments of type 'path' ===
2024/07/17 06:12:23 INFO mlflow.projects.backend.local: === Running command 'source /home/adduser/miniconda3/bin/../etc/profile.d/conda.sh && conda activate mlflow-c1beca2aef7a523d139ba0209b68828b5e298901 1>&2 && python run.py sample1.csv sample.csv raw_data 'Raw file as downloaded'' in run with ID '79de738e1bbb49f5a232b46ae71223a9' ===
wandb: Currently logged in as: samueltwjohnson (samueltwjohnson-Western Governors University). Use `wandb login --relogin` to force relogin
Thread HandlerThread:
Traceback (most recent call last):
File "/home/adduser/miniconda3/envs/mlflow-c1beca2aef7a523d139ba0209b68828b5e298901/lib/python3.10/site-packages/wandb/sdk/internal/internal_util.py", line 49, in run
self._run()
File "/home/adduser/miniconda3/envs/mlflow-c1beca2aef7a523d139ba0209b68828b5e298901/lib/python3.10/site-packages/wandb/sdk/internal/internal_util.py", line 100, in _run
self._process(record)
File "/home/adduser/miniconda3/envs/mlflow-c1beca2aef7a523d139ba0209b68828b5e298901/lib/python3.10/site-packages/wandb/sdk/internal/internal.py", line 280, in _process
self._hm.handle(record)
File "/home/adduser/miniconda3/envs/mlflow-c1beca2aef7a523d139ba0209b68828b5e298901/lib/python3.10/site-packages/wandb/sdk/internal/handler.py", line 136, in handle
handler(record)
File "/home/adduser/miniconda3/envs/mlflow-c1beca2aef7a523d139ba0209b68828b5e298901/lib/python3.10/site-packages/wandb/sdk/internal/handler.py", line 146, in handle_request
handler(record)
File "/home/adduser/miniconda3/envs/mlflow-c1beca2aef7a523d139ba0209b68828b5e298901/lib/python3.10/site-packages/wandb/sdk/internal/handler.py", line 695, in handle_request_run_start
self._system_monitor.probe(publish=True)
File "/home/adduser/miniconda3/envs/mlflow-c1beca2aef7a523d139ba0209b68828b5e298901/lib/python3.10/site-packages/wandb/sdk/internal/system/system_monitor.py", line 175, in probe
k: v for d in [asset.probe() for asset in self.assets] for k, v in d.items()
File "/home/adduser/miniconda3/envs/mlflow-c1beca2aef7a523d139ba0209b68828b5e298901/lib/python3.10/site-packages/wandb/sdk/internal/system/system_monitor.py", line 175, in <listcomp>
k: v for d in [asset.probe() for asset in self.assets] for k, v in d.items()
File "/home/adduser/miniconda3/envs/mlflow-c1beca2aef7a523d139ba0209b68828b5e298901/lib/python3.10/site-packages/wandb/sdk/internal/system/assets/gpu.py", line 365, in probe
info["gpu"] = pynvml.nvmlDeviceGetName(pynvml.nvmlDeviceGetHandleByIndex(0)) # type: ignore
File "/home/adduser/miniconda3/envs/mlflow-c1beca2aef7a523d139ba0209b68828b5e298901/lib/python3.10/site-packages/wandb/vendor/pynvml/pynvml.py", line 1962, in wrapper
return res.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 0: invalid start byte
wandb: ERROR Internal wandb error: file data was not synced
Problem at: /tmp/tmpfxs0kz77/components/get_data/run.py 19 go
Traceback (most recent call last):
File "/home/adduser/miniconda3/envs/mlflow-c1beca2aef7a523d139ba0209b68828b5e298901/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1108, in init
run = wi.init()
File "/home/adduser/miniconda3/envs/mlflow-c1beca2aef7a523d139ba0209b68828b5e298901/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 770, in init
run_start_result = run_start_handle.wait(timeout=30)
File "/home/adduser/miniconda3/envs/mlflow-c1beca2aef7a523d139ba0209b68828b5e298901/lib/python3.10/site-packages/wandb/sdk/lib/mailbox.py", line 271, in wait
raise MailboxError("transport failed")
wandb.errors.MailboxError: transport failed
wandb: ERROR Abnormal program exit
Traceback (most recent call last):
File "/home/adduser/miniconda3/envs/mlflow-c1beca2aef7a523d139ba0209b68828b5e298901/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1108, in init
run = wi.init()
File "/home/adduser/miniconda3/envs/mlflow-c1beca2aef7a523d139ba0209b68828b5e298901/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 770, in init
run_start_result = run_start_handle.wait(timeout=30)
File "/home/adduser/miniconda3/envs/mlflow-c1beca2aef7a523d139ba0209b68828b5e298901/lib/python3.10/site-packages/wandb/sdk/lib/mailbox.py", line 271, in wait
raise MailboxError("transport failed")
wandb.errors.MailboxError: transport failed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/tmp/tmpfxs0kz77/components/get_data/run.py", line 48, in <module>
go(args)
File "/tmp/tmpfxs0kz77/components/get_data/run.py", line 19, in go
run = wandb.init(job_type="download_file")
File "/home/adduser/miniconda3/envs/mlflow-c1beca2aef7a523d139ba0209b68828b5e298901/lib/python3.10/site-packages/wandb/sdk/wandb_init.py", line 1145, in init
raise Exception("problem") from error_seen
Exception: problem
wandb: While tearing down the service manager. The following error has occurred: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/home/adduser/Project-Build-an-ML-Pipeline-Starter/main.py", line 40, in go
_ = mlflow.run(
File "/home/adduser/miniconda3/envs/mlflow-f6548607417935a06d9b14a98cd4db9af6ea5e32/lib/python3.9/site-packages/mlflow/projects/__init__.py", line 345, in run
_wait_for(submitted_run_obj)
File "/home/adduser/miniconda3/envs/mlflow-f6548607417935a06d9b14a98cd4db9af6ea5e32/lib/python3.9/site-packages/mlflow/projects/__init__.py", line 362, in _wait_for
raise ExecutionException("Run (ID '%s') failed" % run_id)
mlflow.exceptions.ExecutionException: Run (ID '79de738e1bbb49f5a232b46ae71223a9') failed
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
2024/07/17 06:12:38 ERROR mlflow.cli: === Run (ID '5f9a379efe9345bd9459b4c4d4a3e9dc') failed ===
Please see below for the packages I am using:
channels:
- conda-forge
- defaults
dependencies: - python=3.9.15
- jupyterlab=3.0.12
- mlflow=2.1.1
- numpy=1.24.1
- cookiecutter=1.7.2
- hydra-core=1.0.6
- matplotlib=3.6.2
- pandas=1.2.3
- git=2.30.2
- pip=20.3.3
- pip:
- wandb==0.13.9
- databricks_cli==0.8.7
Thank you for any help you can provide.