Wandb + ray tune weird logging in dashboard

I am trying to use wandb and ray-tune integration, following the ‘RayTune_with_wandb’ notebook (sorry can only do two links).

My code is here: Google Colab

Currently I’m getting weird behavior where I see runs showing up in the ‘ppo-basic’ project, but with no metrics being logged: Weights & Biases. In addition to those runs, there are also other runs spawned in an ‘uncategorized’ project that seem to contain the metrics I’m logging. Edit: after switching to using tune.Tuner, I see both the ‘empty’ runs and the runs with metrics in the ppo-basic project.

As a separate issue, I had also tried setting the project to an existing project and specifying the group. Similar to above, two sets of runs were spawned, one without metrics, one with. The one with metrics also had a parallel line plots but only showed the hyperparameters that overlapped with ones from runs from another experiment.

Should I be using the examples from the ray docs?

Thanks for any help.

Hi @clabornd, I see a similar issue reported in the past but it appears to have been fixed. I’ll pass the Colab onto our engineering team to take a look at and follow up shortly.

Thank you,

Hi @clabornd, is the code in the Colab you sent still the code you are using? I get an import error when running it as is.

@nathank Apologies, there was a stale import in there: from ray.tune.integration.wandb import wandb_mixin just removed it, hopefully it works now. I just gave it a try and I’m seeing the behavior of runs with no metrics spawning in the project space, and then other runs with metrics getting sent to the ‘uncategorized’ project.

@clabornd I see. Yes, I get the same behavior. I actually get something similar if I run the example Colab you are referring to. However, there are metrics logged in the correct project but they are not the ones logged with wandb.log(). I’ll raise this to our integrations team and follow up shortly.

Hi @clabornd, apologies for the delay on this. This is a known bug and we are currently working on a fix for this. I’ve added you to the ticket so I can follow up here once we have a fix on this.

Thank you,

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.

Dear Daniel,

I am updating you that we have now fixed the logging issue with WandB and Ray tune that you were experiencing and the fixing PR has been merged here https://github.com/wandb/examples/pull/479.

Please let us know if you need anything further and if not I will close this off for you.

Best wishes,