# Resuming run/training **URL:** \ **Category:** W&B Help\ **Tags:** projects, wandb\ **Created:** [May 23, 2022, 10:28pm UTC](https://community.wandb.ai/t/resuming-run-training/2487 "2022-05-23T22:28:02Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![sajmahmo](https://avatars.discourse-cdn.com/v4/letter/s/b19c9b/32.png) [@sajmahmo](https://community.wandb.ai/u/sajmahmo)\ **Post date:** [May 23, 2022, 10:28pm UTC](https://community.wandb.ai/t/resuming-run-training/2487/1 "2022-05-23T22:28:02Z")
Hi (Please note the codes are in italic), I created a new run using code below: _id = wandb.util.generate\_id()_ _run = wandb.init(project=‘checkpoint’, name=‘new\_load’, id=id, config=configs)_ and the results (lets say for 10 epochs) were stored in my account as expected. I also saved the last model in the run using _wandb.save(‘last\_model.h5’)_. Now, I want to continue learning from epoch 10 for 10 more epochs till epoch 20 for the last\_model. So, I first restore the model using the code below: _restored\_model = wandb.restore(‘last\_model.h5’, run\_path="…/checkpoint/id")_ then, I load the weights from restored\_model to the model: _model = build\_model()_ _model.load\_weights(restored\_model.name)_ and then I compiled the model. However, when I execute _model.fit()_, nothing happens, that is the code is executed without any error but there is no training and no epoch just like executing an empty cell. _num\_epoch = config.epochs - wandb.run.step_ _model.fit(x\_train, y\_train, batch\_size=config.batch\_size, verbose=1, epochs=num\_epoch, validation\_data=(x\_valid, y\_valid), shuffle=False, initial\_epoch=wandb.run.step, callbacks=[WandbCallback(training\_data=(x\_train, y\_train), validation\_data=(x\_valid, y\_valid))])_ I really appreciate any help as I am so in need of resuming training. By the way, I have been wondering why in the example below which is in the resume documentation you use model.compile() while loading the entire model. You won’t need compile the model when you load the entire model. I believe it is not correct and you need to edit the code: import keras import numpy as np import wandb from wandb.keras import WandbCallback wandb.init(project=“preemptible”, resume=True) if wandb.run.resumed: # restore the best model **model = keras.models.load\_model(wandb.restore(“model-best.h5”).name)** else: a = keras.layers.Input(shape=(32,)) b = keras.layers.Dense(10)(a) model = keras.models.Model(input=a, output=b) **model.compile(“adam”, loss=“mse”)** model.fit(np.random.rand(100, 32), np.random.rand(100, 10), # set the resumed epoch initial\_epoch=wandb.run.step, epochs=300, # save the best model if it improved each epoch callbacks=[WandbCallback(save\_model=True, monitor=“loss”)]) ---
**Author:** ![ramit\_goolry](https://avatars.discourse-cdn.com/v4/letter/r/85e7bf/32.png) [@ramit\_goolry](https://community.wandb.ai/u/ramit_goolry)\ **Post date:** [May 25, 2022, 10:32pm UTC](https://community.wandb.ai/t/resuming-run-training/2487/2 "2022-05-25T22:32:59Z")
Hi @sajmahmo, I’m sorry this is happening to you - this is very odd behavior. We’ll have to run some tests to get a better sense of the situation here. To start, could you restore the model and call model.evaluate on a hold out set to make sure that the weights are restored correctly? Additionally, it might help to check the value of `num_epoch` that is being sent to `model.fit`. The values of `config.epochs` and `wandb.run.step` might not be aligned. I’ll try to reproduce this issue on my end as well, but it certainly would help to know the results on your end where this issue is known to appear. Thanks, Ramit ---
**Author:** ![sajmahmo](https://avatars.discourse-cdn.com/v4/letter/s/b19c9b/32.png) [@sajmahmo](https://community.wandb.ai/u/sajmahmo)\ **Post date:** [May 26, 2022, 12:50pm UTC](https://community.wandb.ai/t/resuming-run-training/2487/3 "2022-05-26T12:50:18Z")
Hi, I tried what you said and seems the weights are correctly restored. The metric is quite high, so it can’t be weights. `num_epoch` is also fine because I tested before. For example, If I already trained the model with 100 epochs, and I want to continue training to another 100 epochs, I will set `config.epochs=200`, and `wandb.run.step` is equal to 100, so when I pass `num_epoch` to `model.fit()`, it will train the model for 100 more epochs until the epoch 200 (considering that `initial_epoch=wandb.run.step`). I think it must be the argument “resume” in `wandb.init()`. It is somehow confusing whether it should be True or “must” although I tried both, but both resulted in the same issue. I believe resume simply does not match with `model.fit()` because for other things, it works well. For instance, when I wanted to correct the info of configs for a run, I used `resume=True or 'must'`, and it was corrected in the run overview. Thanks for your support, Sajjad ---
**Author:** ![sajmahmo](https://avatars.discourse-cdn.com/v4/letter/s/b19c9b/32.png) [@sajmahmo](https://community.wandb.ai/u/sajmahmo)\ **Post date:** [June 3, 2022, 8:28am UTC](https://community.wandb.ai/t/resuming-run-training/2487/4 "2022-06-03T08:28:08Z")
Dear Ramit, It is been while since your response to my question. I wonder whether you have been working on my problem or what. Please let me know what is going on or it is closed as your point of view. Regards, Sajjad ---
**Author:** ![ramit\_goolry](https://avatars.discourse-cdn.com/v4/letter/r/85e7bf/32.png) [@ramit\_goolry](https://community.wandb.ai/u/ramit_goolry)\ **Post date:** [June 3, 2022, 8:03pm UTC](https://community.wandb.ai/t/resuming-run-training/2487/5 "2022-06-03T20:03:54Z")
Hi Sajjad, Apologies about the delay here. I did look into your problem and tried to reproduce it on my end, but I was not able to reproduce the bug you are seeing. On my end, the restored model seems to train fine and learn correctly. Would it be possible for you to share a google colab with a reproduction of your issue? Thanks, Ramit ---
**Author:** ![sajmahmo](https://avatars.discourse-cdn.com/v4/letter/s/b19c9b/32.png) [@sajmahmo](https://community.wandb.ai/u/sajmahmo)\ **Post date:** [June 5, 2022, 10:51pm UTC](https://community.wandb.ai/t/resuming-run-training/2487/6 "2022-06-05T22:51:19Z")
Hi Ramit, I can’t upload ipynb file here. How can I share the file with you? Regards, Sajjad ---
**Author:** ![ramit\_goolry](https://avatars.discourse-cdn.com/v4/letter/r/85e7bf/32.png) [@ramit\_goolry](https://community.wandb.ai/u/ramit_goolry)\ **Post date:** [June 6, 2022, 5:00pm UTC](https://community.wandb.ai/t/resuming-run-training/2487/7 "2022-06-06T17:00:40Z")
Hey @sajmahmo, You can email us at [support@wandb.com](mailto:support@wandb.com) with the file! Thanks, Ramit ---
**Author:** ![ramit\_goolry](https://avatars.discourse-cdn.com/v4/letter/r/85e7bf/32.png) [@ramit\_goolry](https://community.wandb.ai/u/ramit_goolry)\ **Post date:** [June 9, 2022, 7:39pm UTC](https://community.wandb.ai/t/resuming-run-training/2487/8 "2022-06-09T19:39:11Z")
Hi Sajjad, We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved. Best, Weights & Biases ---
**Author:** ![sajmahmo](https://avatars.discourse-cdn.com/v4/letter/s/b19c9b/32.png) [@sajmahmo](https://community.wandb.ai/u/sajmahmo)\ **Post date:** [June 10, 2022, 11:51am UTC](https://community.wandb.ai/t/resuming-run-training/2487/9 "2022-06-10T11:51:21Z")
Dear Ramit, I sent the email containing the colab file with the title, Resuming run/training. Regards, Sajjad ---
**Author:** ![system](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/1X/366b649231631dbab896843020da0056074ac79d.png) [@system](https://community.wandb.ai/u/system)\ **Post date:** [August 9, 2022, 11:51am UTC](https://community.wandb.ai/t/resuming-run-training/2487/10 "2022-08-09T11:51:38Z")
Closed on August 9, 2022, 11:51am UTC This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.