Logs are are not captured by WANDB when using Celery

W&B Help
1

I have a FastAPI application which heavily uses W&B, celery for queue management, and loguru for logging.
When we don’t use celery, we have the logs in W&B but when we use Celery tasks, we’re experiencing inconsistent log capture by Wandb. Currently:

  • Print statements are being captured and appear in the Wandb output.log file.
  • Loguru logs are not being captured by Wandb.
  • Default Python logging is not being captured by Wandb.

Environment:
wandb version: 0.18.3
python version: 3.11.10
Operating System: MacOS and Ubuntu
Celery version: 5.2.7
Loguru version: 0.7.2

Expected Behavior:
All the stdout (console) logs should appear in wandb → runs > run > logs. but it’s only getting the print statements not default python loggings and loguru logs.

Piece of code that we enqueue experiment:

experiment_json = json.dumps(
    experiment.model_dump(), default=pydantic_encoder
)
enqueue_experiment_with_priority(
    experiment_json, wandb_id, wandb_run_name, user_id
)

def enqueue_experiment_with_priority(
    experiment_json, wandb_id, wandb_run_name, user_id
):
    experiment_data = json.loads(experiment_json)
    queue_name = "default"
    logger.info(f"Enqueueing experiment for wandb_id: {wandb_id} in {queue_name} queue")
    create_celery_experiment.apply_async(
        args=[experiment_json, wandb_id, wandb_run_name, user_id], queue=queue_name
    )

celery = Celery(
    __name__,
    broker=os.environ.get("CELERY_BROKER_URL", "redis://localhost:6379"),
    backend=os.environ.get("CELERY_RESULT_BACKEND", "redis://localhost:6379"),
)
celery.conf.task_default_queue = "default"


@celery.task(name="create_experiments")
def create_celery_experiment(experiment: str, wandb_id, wandb_run_name, user_id):
    logger.info(f"Starting experiment creation for wandb_id: {wandb_id}")
      with ExperimentRunner(
          experiment,
          user_id=user_id,
          wandb_id=wandb_id,
          wandb_name=wandb_run_name,
      ):
          execute(experiment_conf)

class ExperimentRunner:
    def __init__(
        self,
        expr: Experiment,
        user_id: str = "",
        wandb_id: str = "",
        wandb_name: str = "",
    ):
        wandb.init(**something)

Logger is initialized before wandb initialization.
Please let me know if the description is not enough or I should provide any specific details.

Any idea why it’s happening? and possible solutions?

I appreciate your help :pray:

2

This request was closed and merged into request #77273 “[Logs] Inconsistent Log Capture …”.