Is there a way to escape the wandb launch context?

W&B Help
1

I am setting a trigger to launch a job with wandb launch, but what I want to do is log my results to a previous run (I can access the run id within my script). I’ve tested this locally and it all works, but inside the automated job that gets launched I get an error originating from:

wandb.init(id=..., project=..., resume=must)

where it throws:

wandb.errors.UsageError: You provided an invalid value for the `resume` argument. The value 'must' is not a valid option for resuming a run (ivrfb9rk) that does not exist. Please check your inputs and try again with a valid run ID. If you are trying to start a new run, please omit the `resume` argument or use `resume='allow'`.

I also get warnings:

wandb: WARNING Project is ignored when running from wandb launch context. Ignored wandb.init() arg project when running running from launch.
wandb: WARNING Project, entity and id are ignored when running from wandb launch context. Ignored wandb.init() arg id when running running from launch.

so I assume because I’m calling this inside a launch job it is just ignoring the wandb.init command. I’ve even tried launching a new process with python multiprocessing and putting the wandb.init inside there, but I get the same issue. Is there anyway I can escape the launch context and log to the original run I want to in this automated job?

2

Current workaround:

I have to save all the config, then delete all of the WANDB related environment variables (except WANDB_API_KEY), and then call subprocess.call(…) the actual script to run. Not great…

3

Hi @henrycharlesworth91 at this time resuming runs from within Launch isn’t supported. I’ve marked this as a feature request with our Launch team and they will perform tests and will put it in their queue for future work. Will keep you updated once they’ve gotten back to me with next steps/updates