When using wandb for logging every epoch it slows down training from 15s to 1.5Min per epoch

W&B Help
1

Hey,
Just started to use wandb, and somehow when using it to log some basic performance metrics (acc values etc.) after each epoch it slows down my training from appox. 15s per epoch to about 1.5Min per epoch.

Already tried offline mode and wandb.Settings(_disable_stats=True, console=‘off’), as well as forwarding the internal-debug.log (which gets massive) to a null path. All of the approaches did not result in any runtime differences (it still takes 1.5 Min per epoch).

Any hint/help is appreciated.

Best
Paul

2

Hey @pov-hirschhausen, thanks for flagging this! Would you mind sharing a code example so I can take a look and try to reproduce on my end?

3

Hey @pov-hirschhausen, following up here! Would you have any problems with sharing a code example so we can reproduce this on our end?

4

Hi @pov-hirschhausen, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!