How to handle resuming and changing config file

[carla_s]

I have the following scenario:

Let’s say I have started a run with a specific config, e.g. at the beginning of my run I would do something like this

config = {"lr" : 0.01, "a": "b", ..., "sample_interval": 2000}
wandb.init(config = config, ...)

Now after some time, I realize I want to change something about my model. For example, here I want to sample more often. I would then stop the run and rerun my script with the correct resume ID

config = {"lr" : 0.01, "a": "b", ..., "sample_interval": 1000}
wandb.init(config = config, id = OLD_ID, resume = "allow",  ...)

I think the behaviour wandb has, is to then have the config be changed to the second config online. Is the first config just overwritten or can it still be seen somewhere?

In my more concrete usecase, I might want to change more big things. For example: I have trained with one dataloader for some time, then wrote a more efficient dataloader and would like to switch.
Ideally I would want both of the information, the old and the new config to be seen. Is there a way to do this?

[luis_bergua1]

Hi @carla_s, thanks for your question! So you’re right that in the example you provided the config will be overwritten. You can add every config to a different key, this is:

config = {"config_1": {"lr" : 0.01, "a": "b", ..., "sample_interval": 2000}}
run = wandb.init(config = config, ...)
...
run = wandb.init(config = config, id = OLD_ID, resume = "allow",  ...)
run.config["config_2"] = {"lr" : 0.01, "a": "b", ..., "sample_interval": 1000}

However, if you’re changing big things, I would recommend you to use groups so that you create one separate run and you can access those runs individually but also visualize those runs grouped. Please let me know if this helps!

[luis_bergua1]

Hi there, I wanted to follow up on this request. Please let us know if we can be of further assistance or if your issue has been resolved.

[luis_bergua1]

Hi Carla, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!