# Resuming training

**URL:** <https://community.wandb.ai/t/resuming-training/4360>\
**Category:** W&B Help\
**Tags:** wandb\
**Created:** [May 8, 2023, 3:54pm UTC](https://community.wandb.ai/t/resuming-training/4360 "2023-05-08T15:54:11Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![markstent](https://avatars.discourse-cdn.com/v4/letter/m/85f322/32.png) [@markstent](https://community.wandb.ai/u/markstent)\
**Post date:** [May 8, 2023, 3:54pm UTC](https://community.wandb.ai/t/resuming-training/4360/1 "2023-05-08T15:54:11Z")

</div>

Hey everyone, Im new to WandB and would love some advice.

This is my current setup:

1. Run the model first time and save the model every epoch (based on a variable) using the following:

# log wandb artifact

```
            model_artifact = wandb.Artifact(
                f'{args.project_name}',
                type='model',
                description='sonic-diffusion-model-256'
                )
        
            model_artifact.add_dir(args.output_dir)
            wandb.log_artifact(
                model_artifact,
                aliases=[f'step_{global_step}', f'epoch_{epoch}']

```

1. i have resume as ‘True’ in the configs
2. I then load the last saved model (i am using diffusion from hugging face):

if wandb.run.resumed:  
print(“Resuming run…”)  
artifact\_name = args.model\_resume\_name  
artifact = wandb.use\_artifact(artifact\_name)

```
    # Download the model file(s) and return the path to the downloaded artifact
    artifact_dir = artifact.download()

    pipeline = AudioDiffusionPipeline.from_pretrained(artifact_dir)

    mel = pipeline.mel
    model = pipeline.unet

```

How do i continue training from the last epoch i left off from? Is 3) above even necessary? does the resume load the optimizer settings, learning rate at specific epoch?

The docs are not very clear.

I hope i am articulating myself properly.

Mark

---

<div class="post-metadata">

**Author:** ![uma-wandb](https://avatars.discourse-cdn.com/v4/letter/u/b77776/32.png) [@uma-wandb](https://community.wandb.ai/u/uma-wandb)\
**Post date:** [May 10, 2023, 9:42pm UTC](https://community.wandb.ai/t/resuming-training/4360/2 "2023-05-10T21:42:27Z")

</div>

Hi Mark,

I responded to your issue via email shortly ago, but will respond here as well for visibility.

It looks like you’re storing your epochs as aliases, and in order to properly resume training from a given epoch, you need to access that explicitly under your if wandb.run.resumed line. One way you could go about doing that is by including the following line underneath to properly access the correct epoch:

start\_epoch = int(filter(lambda alias: alias.startswith(‘epoch’), artifact.aliases)[0].split(‘\_’)[1])

which would give you the correct epoch to start at going forward.

Let me know if you need anything else!

Uma

---

<div class="post-metadata">

**Author:** ![markstent](https://avatars.discourse-cdn.com/v4/letter/m/85f322/32.png) [@markstent](https://community.wandb.ai/u/markstent)\
**Post date:** [May 11, 2023, 7:16am UTC](https://community.wandb.ai/t/resuming-training/4360/4 "2023-05-11T07:16:10Z")

</div>

Where would i use the start\_epoch once i have it in the training loop…where would i apply this to make it work?

---

<div class="post-metadata">

**Author:** ![markstent](https://avatars.discourse-cdn.com/v4/letter/m/85f322/32.png) [@markstent](https://community.wandb.ai/u/markstent)\
**Post date:** [May 11, 2023, 7:20am UTC](https://community.wandb.ai/t/resuming-training/4360/5 "2023-05-11T07:20:59Z")

</div>

I assume it would be in the training loop and changing it to:

for epoch in range(start\_epoch, args.num\_epochs):

---

<div class="post-metadata">

**Author:** ![uma-wandb](https://avatars.discourse-cdn.com/v4/letter/u/b77776/32.png) [@uma-wandb](https://community.wandb.ai/u/uma-wandb)\
**Post date:** [May 11, 2023, 6:56pm UTC](https://community.wandb.ai/t/resuming-training/4360/6 "2023-05-11T18:56:16Z")

</div>

Hi Mark,

You are definitely correct to assume it would be in the training loop and that you need to change that particular line of code. Since you save the most recent epoch # when you save the artifact, you should be referencing start\_epoch+1 to get the following epoch.

Also, since you are calling scheduler.step() and optimizer.step(), be sure to save those (either as an artifact or anything else of your choosing) to ensure you’re using the correct values when resuming from a specific epoch.

Best,

Uma

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/1X/366b649231631dbab896843020da0056074ac79d.png) [@system](https://community.wandb.ai/u/system)\
**Post date:** [July 10, 2023, 7:21am UTC](https://community.wandb.ai/t/resuming-training/4360/7 "2023-07-10T07:21:47Z")

</div>

Closed on July 10, 2023, 7:21am UTC

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.