Elapsed time per epoch much slower for sweep than for individual runs

W&B Help
1

When I run the same model individually as I would in a sweep, the performance is much better in terms of time elapsed per epoch. In one recent test I saw a 3x improvement (10min vs 30min). I am running the bayes sweep, minimizing the val/loss, and using hyperband with min_iter = 1. Both jobs run on a single A100 40Gb GPU. I have also included the following line as I am running on SLURM:
wandb agent --count 1 SWEEP_ID

2

Hi Noah,

Do you notice this huge difference in performance when you are running Sweep vs a Regular run?

Cheers,
Artsiom

3

Hello Artsiom,

Yes, that is exactly my issue.

Best,
Noah