I am trying to use wandb and ray-tune integration, following the ‘RayTune_with_wandb’ notebook (sorry can only do two links).
My code is here: Google Colab
Currently I’m getting weird behavior where I see runs showing up in the ‘ppo-basic’ project, but with no metrics being logged: Weights & Biases. In addition to those runs, there are also other runs spawned in an ‘uncategorized’ project that seem to contain the metrics I’m logging. Edit: after switching to using tune.Tuner, I see both the ‘empty’ runs and the runs with metrics in the ppo-basic project.
As a separate issue, I had also tried setting the project to an existing project and specifying the group. Similar to above, two sets of runs were spawned, one without metrics, one with. The one with metrics also had a parallel line plots but only showed the hyperparameters that overlapped with ones from runs from another experiment.
Should I be using the examples from the ray docs?
Thanks for any help.
Hi @clabornd, I see a similar issue reported in the past but it appears to have been fixed. I’ll pass the Colab onto our engineering team to take a look at and follow up shortly.
Hi @clabornd, is the code in the Colab you sent still the code you are using? I get an import error when running it as is.
@nathank Apologies, there was a stale import in there:
from ray.tune.integration.wandb import wandb_mixin just removed it, hopefully it works now. I just gave it a try and I’m seeing the behavior of runs with no metrics spawning in the project space, and then other runs with metrics getting sent to the ‘uncategorized’ project.
@clabornd I see. Yes, I get the same behavior. I actually get something similar if I run the example Colab you are referring to. However, there are metrics logged in the correct project but they are not the ones logged with wandb.log(). I’ll raise this to our integrations team and follow up shortly.
Hi @clabornd, apologies for the delay on this. This is a known bug and we are currently working on a fix for this. I’ve added you to the ticket so I can follow up here once we have a fix on this.