Is it possible to log to multiple runs simultaneously

W&B Help
1

I’d like to log to multiple separate runs (experiments) from a single script, but I’m not sure if it’s possible to do this so I’m asking for help here.

The issue I’m facing is that my code uses a relatively small model, but is bottlenecked by data loading so GPU utilization is quite low. Thus, the goal is to update multiple individual models per batch, which would help ‘hide’ the data loading time by spending more time with GPU compute.
Then for each model, I’d have a corresponding ‘run’ object in WandB:

for x, labels in dataloader:
    loss1 = model1(x, labels)
    loss2 = model2(x, labels)
    ...
    lossN = modelN(x, labels)
    run1.log({'loss': loss1.item()})
    run2.log({'loss': loss2.item()})
    ....
    runN.log({'loss': lossN.item()})

The issue is that the traditional wandb.log() seems like a call to some global run instance and can’t be split into multiple individual instances.

Is it possible to achieve this with WandB? Perhaps using wandb.Api() is a starting point, but I’m not sure if it’s feasible.

2

Hello @numpee!

Happy to help!

Running multiple run = wandb.init() at the same time will cause errors or take a long time as the client would have to change runs often so I suggest to log at the end of your script with something like the following:

import wandb

loss1 = []
loss2 = []


### Your training here resulting in
### loss1 = [<loss from run1>]
### loss2 = [<loss from run2>]


run1 = wandb.init(project = <project>)
run1.log({'loss':loss1})
run1.finish()
run2 = wandb.init(project = <project>)
run2.log({'loss':loss2})
run2.finish()

# or log it like

losses = [loss1, loss2]

for loss in losses:
     step = 1
     run = wandb.init(<project-name>)
     for val in loss:
          run.log({"loss": val},  step = step)
          step  +=1 
     run.finish()

This may not be an ideal setup for you but this would work as a workaround to log multiple runs in the same script. Would this work for you?

1 Like
3

Hi, thanks for the response.
I was hoping there’d be a clever way to be able to track values real time in the web client, but I guess not. I think one workaround would be to keep logging one experiment real-time, while others are logged afterwards (like in the second example you provided).
That should work - Thanks for the idea!

4

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.