# Elapsed time per epoch much slower for sweep than for individual runs

**URL:** <https://community.wandb.ai/t/elapsed-time-per-epoch-much-slower-for-sweep-than-for-individual-runs/4061>\
**Category:** W&B Help\
**Tags:** sweeps\
**Created:** [March 15, 2023, 9:45pm UTC](https://community.wandb.ai/t/elapsed-time-per-epoch-much-slower-for-sweep-than-for-individual-runs/4061 "2023-03-15T21:45:58Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![npaulson](https://avatars.discourse-cdn.com/v4/letter/n/c89c15/32.png) [@npaulson](https://community.wandb.ai/u/npaulson)\
**Post date:** [March 15, 2023, 9:45pm UTC](https://community.wandb.ai/t/elapsed-time-per-epoch-much-slower-for-sweep-than-for-individual-runs/4061/1 "2023-03-15T21:45:59Z")

</div>

When I run the same model individually as I would in a sweep, the performance is much better in terms of time elapsed per epoch. In one recent test I saw a 3x improvement (10min vs 30min). I am running the bayes sweep, minimizing the val/loss, and using hyperband with min\_iter = 1. Both jobs run on a single A100 40Gb GPU. I have also included the following line as I am running on SLURM:  
wandb agent --count 1 SWEEP\_ID

---

<div class="post-metadata">

**Author:** ![artsiom](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/artsiom/32/1001_2.png) [@artsiom](https://community.wandb.ai/u/artsiom)\
**Post date:** [March 20, 2023, 7:30pm UTC](https://community.wandb.ai/t/elapsed-time-per-epoch-much-slower-for-sweep-than-for-individual-runs/4061/2 "2023-03-20T19:30:32Z")

</div>

Hi Noah,

Do you notice this huge difference in performance when you are running Sweep vs a Regular run?

Cheers,  
Artsiom

---

<div class="post-metadata">

**Author:** ![npaulson](https://avatars.discourse-cdn.com/v4/letter/n/c89c15/32.png) [@npaulson](https://community.wandb.ai/u/npaulson)\
**Post date:** [March 21, 2023, 5:46pm UTC](https://community.wandb.ai/t/elapsed-time-per-epoch-much-slower-for-sweep-than-for-individual-runs/4061/3 "2023-03-21T17:46:05Z")

</div>

Hello Artsiom,

Yes, that is exactly my issue.

Best,  
Noah

---

<div class="post-metadata">

**Author:** ![artsiom](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/artsiom/32/1001_2.png) [@artsiom](https://community.wandb.ai/u/artsiom)\
**Post date:** [April 2, 2023, 8:26pm UTC](https://community.wandb.ai/t/elapsed-time-per-epoch-much-slower-for-sweep-than-for-individual-runs/4061/4 "2023-04-02T20:26:30Z")

</div>

Could you potentially share a code snippet and we could see if we can reproduce this on our side?

Warmly,  
Artsiom

---

<div class="post-metadata">

**Author:** ![artsiom](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/artsiom/32/1001_2.png) [@artsiom](https://community.wandb.ai/u/artsiom)\
**Post date:** [April 10, 2023, 2:19pm UTC](https://community.wandb.ai/t/elapsed-time-per-epoch-much-slower-for-sweep-than-for-individual-runs/4061/5 "2023-04-10T14:19:47Z")

</div>

Hi Noah,

We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.

Best,  
Weights & Biases

---

<div class="post-metadata">

**Author:** ![artsiom](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/artsiom/32/1001_2.png) [@artsiom](https://community.wandb.ai/u/artsiom)\
**Post date:** [April 13, 2023, 4:04pm UTC](https://community.wandb.ai/t/elapsed-time-per-epoch-much-slower-for-sweep-than-for-individual-runs/4061/6 "2023-04-13T16:04:25Z")

</div>

Hi Noah, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!

---

<div class="post-metadata">

**Author:** ![npaulson](https://avatars.discourse-cdn.com/v4/letter/n/c89c15/32.png) [@npaulson](https://community.wandb.ai/u/npaulson)\
**Post date:** [May 2, 2023, 8:45pm UTC](https://community.wandb.ai/t/elapsed-time-per-epoch-much-slower-for-sweep-than-for-individual-runs/4061/7 "2023-05-02T20:45:04Z")

</div>

Hi artsiom,

Sorry for my slow response. I put together an example to see if I could reproduce the issue, but it doesn’t seem to do so.

I will include it here below in case there is something obvious I can do to stress the system more.

Thanks,  
Noah

> import os  
> import wandb
> 
> from argparse import ArgumentParser
> 
> import torch  
> from torch import nn  
> from torch.nn import functional as F  
> from torch.utils.data import DataLoader  
> from torch.utils.data import random\_split  
> from torchvision.datasets import MNIST  
> from torchvision import transforms  
> import pytorch\_lightning as pl
> 
> class LitAutoEncoder(pl.LightningModule):  
> def **init** (self):  
> super(). **init** ()  
> self.encoder = nn.Sequential(  
> nn.Linear(28 \* 28, 64),  
> nn.ReLU(),  
> nn.Linear(64, 3))  
> self.decoder = nn.Sequential(  
> nn.Linear(3, 64),  
> nn.ReLU(),  
> nn.Linear(64, 28 \* 28))
> 
> ```
> def forward(self, x):
> embedding = self.encoder(x)
> return embedding
> 
> def configure_optimizers(self):
> optimizer = torch.optim.Adam(self.parameters(), lr=1e-3)
> return optimizer
> 
> def training_step(self, train_batch, batch_idx):
> x, y = train_batch
> x = x.view(x.size(0), -1)
> z = self.encoder(x)
> x_hat = self.decoder(z)
> loss = F.mse_loss(x_hat, x)
> self.log('train_loss', loss)
> wandb.log({"train_loss": loss})
> wandb.log({"epoch": self.current_epoch})
> return loss
> 
> def validation_step(self, val_batch, batch_idx):
> x, y = val_batch
> x = x.view(x.size(0), -1)
> z = self.encoder(x)
> x_hat = self.decoder(z)
> loss = F.mse_loss(x_hat, x)
> self.log('val_loss', loss)
> wandb.log({"val_loss": loss})
> 
> ```
> 
> def main(args):
> 
> ```
> project = os.getenv("TEST_WANDB_PROJ")
> entity = os.getenv("TEST_WANDB_ACCT")
> print(f'project: {project}, entity: {entity}')
> 
> log_dir = os.getenv("TEST_LOG_DIR")
> if log_dir is None:
> log_dir = "./data/TEST_LOG_DIR"
> print(
> "Using default wandb log dir path of ./data/TEST_LOG_DIR. This can be adjusted with the environment variable `TEST_LOG_DIR`"
> )
> if not os.path.exists(log_dir):
> os.makedirs(log_dir)
> assert (
> project is not None and entity is not None
> ), "Please set environment variables `TEST_WANDB_ACCT` and `TEST_WANDB_PROJ` with \n\
> your wandb user/organization name and project title, respectively."
> experiment = wandb.init(
> project=project,
> entity=entity,
> config=args,
> dir=log_dir,
> reinit=True,
> )
> config = wandb.config
> wandb.run.name = args.run_name
> wandb.run.save()
> 
> # data
> dataset = MNIST('', train=True, download=True, transform=transforms.ToTensor())
> mnist_train, mnist_val = random_split(dataset, [55000, 5000])
> 
> train_loader = DataLoader(mnist_train, batch_size=args.batch_size)
> val_loader = DataLoader(mnist_val, batch_size=args.batch_size)
> 
> # model
> model = LitAutoEncoder()
> 
> logger = pl.loggers.WandbLogger(
> experiment=experiment, save_dir="./data/TEST_LOG_DIR")
> logger.log_hyperparams(config)
> 
> # training
> trainer = pl.Trainer(gpus=1, num_nodes=1, precision=32, limit_train_batches=0.5, max_epochs=50)
> trainer.fit(model, train_loader, val_loader)
> 
> experiment.finish()
> 
> ```
> 
> if **name** == ‘ **main** ’:
> 
> ```
> parser = ArgumentParser()
> parser.add_argument("--run_name", type=str, required=True)
> parser.add_argument("--batch_size", type=int, default=32)
> 
> args = parser.parse_args()
> main(args)
> 
> ```

---

<div class="post-metadata">

**Author:** ![artsiom](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/artsiom/32/1001_2.png) [@artsiom](https://community.wandb.ai/u/artsiom)\
**Post date:** [May 15, 2023, 3:28pm UTC](https://community.wandb.ai/t/elapsed-time-per-epoch-much-slower-for-sweep-than-for-individual-runs/4061/8 "2023-05-15T15:28:42Z")

</div>

Hi Noah,

Trying to reproduce this and still no luck on my side, could you send me a link to the workspace of a sweep and a regular run for comparison where one runs slower than the other?

Warmly,  
Artsiom

---

<div class="post-metadata">

**Author:** ![artsiom](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/artsiom/32/1001_2.png) [@artsiom](https://community.wandb.ai/u/artsiom)\
**Post date:** [May 18, 2023, 5:17pm UTC](https://community.wandb.ai/t/elapsed-time-per-epoch-much-slower-for-sweep-than-for-individual-runs/4061/9 "2023-05-18T17:17:18Z")

</div>

Hi Noah,

We wanted to follow up with you regarding your support request as we have not heard back from you.

Best,  
Weights & Biases

---

<div class="post-metadata">

**Author:** ![npaulson](https://avatars.discourse-cdn.com/v4/letter/n/c89c15/32.png) [@npaulson](https://community.wandb.ai/u/npaulson)\
**Post date:** [May 22, 2023, 2:49pm UTC](https://community.wandb.ai/t/elapsed-time-per-epoch-much-slower-for-sweep-than-for-individual-runs/4061/10 "2023-05-22T14:49:21Z")

</div>

Hi Artisiom,

I ran a recent example and I cannot recreate the issue any more. I don’t have any explanation why. Thank you for the help - perhaps it is best if we close the question for now.

Thanks,  
Noah

---

<div class="post-metadata">

**Author:** ![artsiom](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/artsiom/32/1001_2.png) [@artsiom](https://community.wandb.ai/u/artsiom)\
**Post date:** [May 23, 2023, 7:22pm UTC](https://community.wandb.ai/t/elapsed-time-per-epoch-much-slower-for-sweep-than-for-individual-runs/4061/11 "2023-05-23T19:22:16Z")

</div>

No problem!

I am very glad this has been solved. 🙂

Cheers,  
Artsiom

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/1X/366b649231631dbab896843020da0056074ac79d.png) [@system](https://community.wandb.ai/u/system)\
**Post date:** [July 21, 2023, 2:50pm UTC](https://community.wandb.ai/t/elapsed-time-per-epoch-much-slower-for-sweep-than-for-individual-runs/4061/12 "2023-07-21T14:50:01Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.