Currently I’m getting weird behavior where I see runs showing up in the ‘ppo-basic’ project, but with no metrics being logged: Weights & Biases. In addition to those runs, there are also other runs spawned in an ‘uncategorized’ project that seem to contain the metrics I’m logging. Edit: after switching to using tune.Tuner, I see both the ‘empty’ runs and the runs with metrics in the ppo-basic project.
As a separate issue, I had also tried setting the project to an existing project and specifying the group. Similar to above, two sets of runs were spawned, one without metrics, one with. The one with metrics also had a parallel line plots but only showed the hyperparameters that overlapped with ones from runs from another experiment.
Hi @clabornd, I see a similar issue reported in the past but it appears to have been fixed. I’ll pass the Colab onto our engineering team to take a look at and follow up shortly.
@nathank Apologies, there was a stale import in there: from ray.tune.integration.wandb import wandb_mixin just removed it, hopefully it works now. I just gave it a try and I’m seeing the behavior of runs with no metrics spawning in the project space, and then other runs with metrics getting sent to the ‘uncategorized’ project.
@clabornd I see. Yes, I get the same behavior. I actually get something similar if I run the example Colab you are referring to. However, there are metrics logged in the correct project but they are not the ones logged with wandb.log(). I’ll raise this to our integrations team and follow up shortly.
Hi @clabornd, apologies for the delay on this. This is a known bug and we are currently working on a fix for this. I’ve added you to the ticket so I can follow up here once we have a fix on this.
I am updating you that we have now fixed the logging issue with WandB and Ray tune that you were experiencing and the fixing PR has been merged here https://github.com/wandb/examples/pull/479.
Please let us know if you need anything further and if not I will close this off for you.