Wandb error by usage of mlflow and hydra regarding protobuf lib

W&B Help
1

Hi all,
having an issue and need some help for Python project inside a Udacity course.
We shall use Python 3.8, therefore I am using 3.8.16 and having created a virtual env starting from that version and activated virtual env.

Using Jupyter Lab I am trying to use a pipeline for mlflow, hydra and wandb I am getting the same error as mentioned e.g. in

I tried already the mentioned protobuf version 3.20.0 and 3.20.1 and different wandb versions via conda.yml, always getting the same error:

name: download_data
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.8
  - requests=2.24.0
  - pip=20.3.3
  - mlflow=2.1.1
  - hydra-core=1.3.1      #1.0.6
#  - wandb=0.13.10
  - pip:
      #- wandb==0.10.21
      - protobuf==3.20.0
      - wandb==0.12.2
      - hydra-joblib-launcher==1.1.2

Still getting the following stacktrace for the ML model and its param setting:
:sun_with_face: :x: mlflow run . -P hydra_options=“random_forest_pipeline.random_forest.max_depth=5”
2023/02/12 17:55:15 INFO mlflow.utils.conda: === Creating conda environment mlflow-8284cfd5101c5c151da499d35f932662f514265c ===
Collecting package metadata (repodata.json): …working… done
Solving environment: …working… done
Preparing transaction: …working… done
Verifying transaction: …working… done
Executing transaction: …working… done
Installing pip dependencies: …working… done
2023/02/12 18:02:28 INFO mlflow.projects.utils: === Created directory /tmp/tmpcv9ww8ry for downloading remote URIs passed to arguments of type ‘path’ ===
2023/02/12 18:02:28 INFO mlflow.projects.backend.local: === Running command ‘source /home/ilona/miniconda3/bin/…/etc/profile.d/conda.sh && conda activate mlflow-8284cfd5101c5c151da499d35f932662f514265c 1>&2 && python main.py $(echo random_forest_pipeline.random_forest.max_depth=5)’ in run with ID ‘a6d323399b9242c1b227bae18109b69d’ ===
2023/02/12 18:02:39 INFO mlflow.utils.conda: Conda environment mlflow-70e71a7afdc413046f59e7b87abd03dc50d8745f already exists.
2023/02/12 18:02:39 INFO mlflow.projects.utils: === Created directory /tmp/tmpsxd_dtnx for downloading remote URIs passed to arguments of type ‘path’ ===
2023/02/12 18:02:39 INFO mlflow.projects.backend.local: === Running command ‘source /home/ilona/miniconda3/bin/…/etc/profile.d/conda.sh && conda activate mlflow-70e71a7afdc413046f59e7b87abd03dc50d8745f 1>&2 && python run.py --train_data exercise_6/data_train.csv:latest
–model_config /home/ilona/MLOps/nd0821-c2-build-model-workflow-exercises/lesson-4-training-validation-experiment-tracking/exercises/exercise_11/starter/outputs/2023-02-12/18-02-33/random_forest_config.yml’ in run with ID ‘59002ca997e54d1dac9fd30e5a418009’ ===
Traceback (most recent call last):
File “run.py”, line 15, in
import wandb
File “/home/ilona/miniconda3/envs/mlflow-70e71a7afdc413046f59e7b87abd03dc50d8745f/lib/python3.8/site-packages/wandb/init.py”, line 38, in
from wandb import sdk as wandb_sdk
File “/home/ilona/miniconda3/envs/mlflow-70e71a7afdc413046f59e7b87abd03dc50d8745f/lib/python3.8/site-packages/wandb/sdk/init.py”, line 12, in
from .wandb_init import init # noqa: F401
File “/home/ilona/miniconda3/envs/mlflow-70e71a7afdc413046f59e7b87abd03dc50d8745f/lib/python3.8/site-packages/wandb/sdk/wandb_init.py”, line 29, in
from .backend.backend import Backend
File “/home/ilona/miniconda3/envs/mlflow-70e71a7afdc413046f59e7b87abd03dc50d8745f/lib/python3.8/site-packages/wandb/sdk/backend/backend.py”, line 17, in
from …interface import interface
File “/home/ilona/miniconda3/envs/mlflow-70e71a7afdc413046f59e7b87abd03dc50d8745f/lib/python3.8/site-packages/wandb/sdk/interface/interface.py”, line 18, in
from wandb.proto import wandb_internal_pb2 as pb
File “/home/ilona/miniconda3/envs/mlflow-70e71a7afdc413046f59e7b87abd03dc50d8745f/lib/python3.8/site-packages/wandb/proto/wandb_internal_pb2.py”, line 15, in
from wandb.proto import wandb_telemetry_pb2 as wandb_dot_proto_dot_wandb__telemetry__pb2
File “/home/ilona/miniconda3/envs/mlflow-70e71a7afdc413046f59e7b87abd03dc50d8745f/lib/python3.8/site-packages/wandb/proto/wandb_telemetry_pb2.py”, line 34, in
_descriptor.FieldDescriptor(
File “/home/ilona/miniconda3/envs/mlflow-70e71a7afdc413046f59e7b87abd03dc50d8745f/lib/python3.8/site-packages/google/protobuf/descriptor.py”, line 560, in new
_message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:

  1. Downgrade the protobuf package to 3.20.x or lower.
  2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: Changes made on May 6, 2022 | Protocol Buffers Documentation
…

In advance, thank you very much for help and solution proposals,
Ilona

2

Hey Ilona,

Thank you for contacting us and bringing this up.
I have tried reproducing it but don’t get the same error.

To clear up any misunderstandings I wish to know if you are using virtualenv package in python and/or conda envs? Mixing up both may cause import issues in general.
Could you check if you have a virtualenv in conda env or vice versa? Let me know if that is not the case.

Also the error message from the issue sent isn’t exactly the same though it is the same cause.
Does the error happen just when you are trying to import wandb or without?

3

Hello Ilona,

Following up to know if you have tried what I mentioned?

4

Hello Ilona,

I will be closing this ticket for lack of response. If you encounter the same issue or another one feel free to contact us.

5

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.