Say I have some Python script sweep.py that runs a lengthy hyperparameter sweep. Under the hood, sweep.py uses many other Python files to train and evaluate the models.
Now, say that after starting the sweep, I want to edit one of the Python files (that are used by the sweep), but I don’t want to stop the running sweep, and I don’t care that it uses the “older” version of the files. This can happen, for example, if I want to add some support to a new use case that requires changing the files, but I don’t want to lose the current running sweep. So, I was wondering if there’s any harm in editing the files while a sweep is running? Can this cause any problems? And what will happen in this case? What I want to happen is that the sweep that has started will run till completion with the older version files (those it started with) and that the editing will only affect newer sweeps that start after the editing (this, for example, what would happen if you run a python script and just edit the script after you started running it – the editing won’t affect the old run), however, I don’t know how W&B run the sweep under the hood?
In terms of editing one of the files that the script will use, this will not break the sweep and the sweep will continue to run with the different file (as long as it is the same name). Sweeps will create runs with different hyperparameters so changing the contents of the files would act the same as changing the files between or during a run (depending on when you change the content of the file).
Thanks for the reply, but I’m not sure I understand.
Say that at time t1 I start a sweep that is supposed to last until t2 and uses some file train.py.
At some time t3 < t2 but t3 > t1, I change the file train.py while the sweep is still running.
What will happen exactly? will that break the sweep? will the sweep continue and will use the older version of train.py?
The files that train.py references (such as other scripts, let’s say a helper.py with helper functions is imported in train.py)
If you change train.py during a sweep, the sweep will continue to use the original version of train.py. It will not use the updated version of train.py.
However, if you change helper.py during the sweep, the sweep will use the newer version of helper.py.