Run Time is inaccurate because of including upload time

W&B Help
1

Some runs will spend minutes because of my terrible network:

wandb: Waiting for W&B process to finish... (success).

I find the Run Time column in UI will also contain the uploading time (by comparing with other runs’ Run Time).

My script is organized as follow:

def main(config):
    ...
    wandb.init(wandb_config)
    ...

if __name__ == '__main__':
    config = blabla
    for p in [p1, p2, p3]:  # for loop to tune hyperparameters
        config.param = p
        main(config)

To fix this issue, should I use wandb.finish in the end of the main() function? As the doc of wandb.finish lists:

Marks a run as finished, and finishes uploading all data.

I worry about whether this func will kill my slow data uploading worker.

Or any other solutions?

2

I find the Run Time is also inaccurate when sync the offline-run instance. After sync, the Run-Time is about 1 min (seems to be the time spent on the sync command).

3

Hi Yago!

Looking into this.

Cheers!
Artsiom

4

Hi Yao!

Sorry for the late response. I have consulted with a few others on my team and this seems to be a bug, considering the fact that you still get the wrong time using the offline mode.

Would you send us some info on how to reproduce this so we can fix it asap?

Cheers!
Artsiom

5

I think reproduce this issue is easy.

  1. Run wandb in offline mode.
  2. After the run is finish, use wandb sync run_path command to upload the run.
  3. The RunTime colum is several seconds (time the sync command spent) in the UI of uploaded run.

My test code is like:

os.environ['WANDB_MODE'] = 'offline'
run = wandb.init(...)
...
run.finish()
6

Hi Yao!

Thank you for the directions. I was able to reproduce it and will send it over to the engineers!
The only real workaround for this is recording the run-times yourself and then logging them to wandb.

Artsiom

7

Hi Yao,

We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.

Best,
Weights & Biases

8

Hi Yao, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!

9

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.