I have the following scenario:
Let’s say I have started a run with a specific config, e.g. at the beginning of my run I would do something like this
config = {"lr" : 0.01, "a": "b", ..., "sample_interval": 2000}
wandb.init(config = config, ...)
Now after some time, I realize I want to change something about my model. For example, here I want to sample more often. I would then stop the run and rerun my script with the correct resume ID
config = {"lr" : 0.01, "a": "b", ..., "sample_interval": 1000}
wandb.init(config = config, id = OLD_ID, resume = "allow", ...)
I think the behaviour wandb has, is to then have the config be changed to the second config online. Is the first config just overwritten or can it still be seen somewhere?
In my more concrete usecase, I might want to change more big things. For example: I have trained with one dataloader for some time, then wrote a more efficient dataloader and would like to switch.
Ideally I would want both of the information, the old and the new config to be seen. Is there a way to do this?