# Hyperparameter tuning combined with k-fold cross validation **URL:** \ **Category:** W&B Help\ **Tags:** sweeps\ **Created:** [February 14, 2023, 3:16pm UTC](https://community.wandb.ai/t/hyperparameter-tuning-combined-with-k-fold-cross-validation/3881 "2023-02-14T15:16:52Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![mbp](https://avatars.discourse-cdn.com/v4/letter/m/4491bb/32.png) [@mbp](https://community.wandb.ai/u/mbp)\ **Post date:** [February 14, 2023, 3:16pm UTC](https://community.wandb.ai/t/hyperparameter-tuning-combined-with-k-fold-cross-validation/3881/1 "2023-02-14T15:16:52Z")
I found [this official example](https://github.com/wandb/examples/blob/master/examples/wandb-sweeps/sweeps-cross-validation/train-cross-validation.py) showcasing an implementation of k-fold cross validation using Sweeps. However, I am doing hyperparameter tuning with sweeps so I am coming from a different angle: I want to do k-fold CV for one given set of parametersr for each sweep run. I would imagine that there should be sub-groups for each CV-group in the sweep view of the web interface. Is this possible to do with Wandb or should I look elsewhere? Thank you! EDIT: the rationale behind it is to prevent optimizing hyper parameters to overfit the test set. If you have another means to reach this goal, I am open for it. ---
**Author:** ![bill-morrisson](https://avatars.discourse-cdn.com/v4/letter/b/90ced4/32.png) [@bill-morrisson](https://community.wandb.ai/u/bill-morrisson)\ **Post date:** [February 16, 2023, 8:17pm UTC](https://community.wandb.ai/t/hyperparameter-tuning-combined-with-k-fold-cross-validation/3881/2 "2023-02-16T20:17:45Z")
Hey, Thank you for contacting us! Yes, it is possible to perform k-fold cross-validation for a given set of hyperparameters with Wandb Sweeps. In fact, the example you found is a good starting point for implementing k-fold cross-validation in your own Sweep runs. ---
**Author:** ![mbp](https://avatars.discourse-cdn.com/v4/letter/m/4491bb/32.png) [@mbp](https://community.wandb.ai/u/mbp)\ **Post date:** [February 20, 2023, 4:52pm UTC](https://community.wandb.ai/t/hyperparameter-tuning-combined-with-k-fold-cross-validation/3881/3 "2023-02-20T16:52:27Z")
Thank you for your reply. Maybe I misunderstood, but as I understand it, the example that I found does not perform hyperparameter tuning at all and instead just performs k-fold CV using Sweep runs. I.e., the example uses one sweep run for each fold resulting in a total of k runs. Can you confirm that? Additionally, the example uses multiprocessing for some reason while at the same time `join`ing each of the created processes immediately (see [here](https://github.com/wandb/examples/blob/5727f44e4aa1acc995a8b8c63f44c3266203c165/examples/wandb-sweeps/sweeps-cross-validation/train-cross-validation.py#L84)). In my understanding, that means that each process runs after the previous one (no parallelism) and the usage of threading seems to be due to other non-obvious reasons. The problem for me seems to be that with a given set of hyperparameters, each call to `wandb.init()` refers to the very same run internally. So if I want to loop over the folds using the same hyperparameters, Wandb ends up overwriting the previous runs/folds every time. Here is a minimal working example: ```auto import wandb import wandb.sdk import randomname import numpy as np from sklearn.model_selection import KFold SWEEP_CONFIG = { "method": "random", "name": "my_config", "metric": {"goal": "minimize", "name": "val_root_mean_squared_error"}, "parameters": { "param1": {"values": [8, 16, 32]}, "param2": {"values": [1, 2, 4]}, }, } class Experiment: def __init__ (self) -> None: self.x_train = np.random.random((2048, 3, 1)) self.y_train = np.random.random((2048, 1)) def train(self) -> None: kf = KFold(n_splits=4, shuffle=True) cv_name = randomname.get_name() for fold, (ix_train, ix_val) in enumerate(kf.split(self.x_train)): x_fold_train, y_fold_train = self.x_train[ix_train], self.y_train[ix_train] x_fold_val, y_fold_val = self.x_train[ix_val], self.y_train[ix_val] run_name = f"{cv_name}-{fold:02}" run = wandb.init(group=f"cv_{cv_name}", name=run_name, reinit=True) assert run is not None assert type(run) is wandb.sdk.wandb_run.Run wandb.summary["cv_fold"] = fold wandb.summary["num_cv_folds"] = kf.n_splits wandb.summary["cv_random_state"] = kf.random_state param1 = wandb.config.param1 param2 = wandb.config.param2 # random result for MWE rmse = param1 * np.mean(y_fold_train) + param2 * np.mean(y_fold_val) score = rmse wandb.log({"val_root_mean_squared_error": score}) wandb.finish() if __name__ == " __main__": exp = Experiment() sweep_id = wandb.sweep(sweep=SWEEP_CONFIG, project="my_proj") wandb.agent( sweep_id=sweep_id, function=exp.train, project="my_proj", # count=40, ) ``` Could you point out what needs to change in this example for it to work? EDIT: Obviously I could just NOT log the training to Wandb and instead only return the average result score for all folds - however, this is not what I want. I want to be able to compare the loss graphs of different folds etc. ---
**Author:** ![mbp](https://avatars.discourse-cdn.com/v4/letter/m/4491bb/32.png) [@mbp](https://community.wandb.ai/u/mbp)\ **Post date:** [February 28, 2023, 2:47pm UTC](https://community.wandb.ai/t/hyperparameter-tuning-combined-with-k-fold-cross-validation/3881/4 "2023-02-28T14:47:44Z")
I just want to make sure I understand correctly: Even though you said that this should be possible to do, I have shown in my MVP that it does not work. In my understanding this means that you did misunderstand what I meant (or didn’t know) and it is indeed not possible to perform hyperparameter tuning in addition to k-fold CV with W&B and I will have to look somewhere else. Could you confirm this? Please let me know. ---
**Author:** ![thecml](https://avatars.discourse-cdn.com/v4/letter/t/7c8e57/32.png) [@thecml](https://community.wandb.ai/u/thecml)\ **Post date:** [March 3, 2023, 2:21pm UTC](https://community.wandb.ai/t/hyperparameter-tuning-combined-with-k-fold-cross-validation/3881/6 "2023-03-03T14:21:58Z")
I’m also looking for ways to do this. As @mbp wrote, it would be nice to have metric curves per run and then be able to group these per fold. Now, if you just create a new run per every fold, it gets overwritten next configuration sweep. ---
**Author:** ![bill-morrisson](https://avatars.discourse-cdn.com/v4/letter/b/90ced4/32.png) [@bill-morrisson](https://community.wandb.ai/u/bill-morrisson)\ **Post date:** [March 3, 2023, 2:48pm UTC](https://community.wandb.ai/t/hyperparameter-tuning-combined-with-k-fold-cross-validation/3881/7 "2023-03-03T14:48:40Z")
Hey @MBP, Thank you very much for your patience! I mentioned that the example you found was a good starting point for implementing k-fold cross validation in your sweep runs. Meaning it was a good starting point to add the sweep configurations to launch the agents. The example you provided do run one sweep for each fold. It seems like I didn’t get to understand what you wanted. Are you trying to parallelize sweep agents in such a way that each sweep agent performs k-fold CV? ---
**Author:** ![mbp](https://avatars.discourse-cdn.com/v4/letter/m/4491bb/32.png) [@mbp](https://community.wandb.ai/u/mbp)\ **Post date:** [March 4, 2023, 8:57pm UTC](https://community.wandb.ai/t/hyperparameter-tuning-combined-with-k-fold-cross-validation/3881/8 "2023-03-04T20:57:29Z")
Hi @bill-morrisson, I am not sure what more I can do to explain this. I have even added source code above so that you can run it yourself and see the issue. I will try to rephrase it: I want to run a hyperparameter study with W&B and I want to use Sweeps for it. This is well-documented and works on its own. For each `run` I will receive a set of parameters from `run.config`. So far so good! Now I want to use one `run` and the parameters from this particular `run` and I want to perform k-fold cross validation with these parameters. That would be easy - I just need to run the training in a loop, and train one model for each fold, right? But now, I want to log all those runs to W&B as well! How to do that? It seems it’s not possible because when I use the `wandb.log` and other functions in the loop, the previous value will just be overwritten. This is the problem I want to solve - how to do this without overwriting the previous values? Additionally, one could think that if I run `wandb.init` once for each fold, then maybe the `wandb.log` will not be overwritten and instead the folds will be logged. Alas the values will still be overwritten. A new `run` will be created in W&B but the previous run will just disappear. For example the first run/fold is called `mysterious-sweep-1` and then the second fold will start, a new `run` with the name `epic-sweep-2`. And now, the run `mysterious-sweep-1` is disappeared completely and all previous logged values are overwritten. I hope this helps to clarify. ---
**Author:** ![magenbrot](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@magenbrot](https://community.wandb.ai/u/magenbrot)\ **Post date:** [March 8, 2023, 7:57pm UTC](https://community.wandb.ai/t/hyperparameter-tuning-combined-with-k-fold-cross-validation/3881/9 "2023-03-08T19:57:25Z")
I understand your question. I jsut want to point oout, I’m having the same issue. Previous runs getting overwritten. Im ending up with only 3 run per job, not 1 per kFold. When I run it with kfold not within a sweep agent, it works as inteden (grouped in the UI, all there) Anyone any idea? ---
**Author:** ![galharari](https://avatars.discourse-cdn.com/v4/letter/g/e99b99/32.png) [@galharari](https://community.wandb.ai/u/galharari)\ **Post date:** [March 11, 2023, 4:20pm UTC](https://community.wandb.ai/t/hyperparameter-tuning-combined-with-k-fold-cross-validation/3881/10 "2023-03-11T16:20:45Z")
I have the same problem where multiple runs for the same group, each run for a different fold, are all overwritten into 1 run for the entire k-folds. It means that **all the information** of the runs not including the last fold run **is lost**. ---
**Author:** ![mbp](https://avatars.discourse-cdn.com/v4/letter/m/4491bb/32.png) [@mbp](https://community.wandb.ai/u/mbp)\ **Post date:** [March 14, 2023, 9:05am UTC](https://community.wandb.ai/t/hyperparameter-tuning-combined-with-k-fold-cross-validation/3881/11 "2023-03-14T09:05:25Z")
I think we don’t get any official replies because either this is - not possible and therefore we wait in vain, or - it is so simple and obvious that we are being ignored. I hope it’s the latter and we can find out how to do it ourselves. 😅 ---
**Author:** ![bill-morrisson](https://avatars.discourse-cdn.com/v4/letter/b/90ced4/32.png) [@bill-morrisson](https://community.wandb.ai/u/bill-morrisson)\ **Post date:** [March 14, 2023, 1:15pm UTC](https://community.wandb.ai/t/hyperparameter-tuning-combined-with-k-fold-cross-validation/3881/12 "2023-03-14T13:15:15Z")
Hi @MBP, Sorry for the time taken to get back to you. We haven’t yet dug into it specifically. We’ll be looking into it with our engineering team and let you know. ---
**Author:** ![magenbrot](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@magenbrot](https://community.wandb.ai/u/magenbrot)\ **Post date:** [March 15, 2023, 11:17am UTC](https://community.wandb.ai/t/hyperparameter-tuning-combined-with-k-fold-cross-validation/3881/13 "2023-03-15T11:17:16Z")
I have posted an github issue. I think they are referring to that. > > > \### Describe the bug > > > Using KFold nested runs, within a sweep loop. Runs are g…rouped with a random id. But after the sweep / all folds are finished, only the most recent KFold run is persistent and old data is overwritten. Every KFold, there is a new run created, but ID and Path are always the same. Only in the next sweep run the group cand run id changes. > > You can see, the printed run id in the attached log and also the "view run at" URL stays identical. > I experimented with reinit=True/False, adding extra wandb.finish(), but no change. > When calling the run\_loop() manually, it gives the expected results. The issue is only when called from sweep agent. > Using notebook or .py file for sweeps changes nothing. I tried aswell, not to use "with wand.init() as run", but start and save the run object and use it then with run.finish() > > > \`\`\`python > class Training(): > def \_\_init\_\_(self, base\_config, device): > self.base\_config = base\_config > self.device = device > self.datalist = None > > def set\_data(self, inputs, outputs): > self.datalist = \[(inputs\[i\], outputs\[i\]) for i in range(len(inputs))\] > > def train\_loop(self, kFoldSplits=2): > myKfold = KFold(n\_splits=kFoldSplits, shuffle=True) > kSplit = 0 > for train\_index, val\_index in myKfold.split(self.datalist): > kSplit += 1 > with wandb.init(config=self.base\_config, group=self.base\_config\['group\_name'\]) as temp\_run: # New run for every fold > self.run\_config = temp\_run.config # Updated config to reflect sweep changes > print("run id: ", temp\_run.id) # ISSUE here: for every kFOld iteration, the new init produces same ID and name, overwriting older folds > print("run name: ", temp\_run.name) > print("run path: ", temp\_run.dir) > print(f"Start KFold {kSplit}") > #trainloader, validationloader = self.get\_dataloader(train\_index, val\_index) > for epoch in range(self.run\_config\['epochs'\]): > print("-"\*25) > print(f"Start Training Epoch {epoch}") > #self.train\_epoch(trainloader, temp\_run) # Real training would be here > temp\_run.log({"target": epoch}) # obviously bogus, just to test > print(f"Finish KFold {kSplit}") > print("$"\*80) > > \########## > > def run\_job(): # Main method, called once per sweep > device = torch.device("cuda:0" if torch.cuda.is\_available() else "cpu") > group\_name=f"experiment\_{random.randrange(111111, 999999, 6)}" # Random group name, unique for each sweep job > print("Group name: ", group\_name) > base\_config = { > "lr": 0.001, > "epochs": 10, > "batch\_size": 32, > "kfold": 3, # ... > "group\_name": group\_name, > } # My config which contains lr, epochs, batch\_size, etc. > > training = Training(base\_config = base\_config, device = device) # Training wrapper - Stripped version > training.set\_data(torch.randn(100, 10),torch.randn(100, 1)) > training.train\_loop(kFoldSplits=3) # start the training with folds > > \########### Sweep creation and start > > sweep\_configuration = { > 'method': 'random', > 'name': 'sweep\_test', > 'metric': {'goal': 'maximize', 'name': 'target'}, > 'parameters': > { > 'epochs': {'values': \[2, 4\]}, > 'kfold': {'values': \[2, 4\]}, > } > } > START\_NEW\_SWEEP = True > > if START\_NEW\_SWEEP: > sweep\_id = wandb.sweep(sweep=sweep\_configuration, project="test-sweep\_project\_name") > print(sweep\_id) > > \#sweep\_id = sweep\_id > sweep\_id = "pp7qjzz9" > wandb.agent(sweep\_id, function=run\_job, project="test-sweep\_project\_name") > > \## Running the run\_job with sweep agent, it produices the opverwriting issue > \## Calling run\_job individually, produces expected resutls, one group, with num\_fold individually tracked runs > \`\`\` > > > \`\`\`shell > > wandb: Agent Starting Run: a0jp1107 with config: > wandb: epochs: 4 > wandb: kfold: 4 > Group name: experiment\_165261 > wandb: Currently logged in as: magenbrot. Use \`wandb login --relogin\` to force relogin > Tracking run with wandb version 0.13.11 > Syncing run lilac-sweep-3 to Weights & Biases (docs) > Sweep page: https://wandb.ai/magenbrot/test-sweep\_project\_name/sweeps/pp7qjzz9 > View project at https://wandb.ai/magenbrot/test-sweep\_project\_name > View sweep at https://wandb.ai/magenbrot/test-sweep\_project\_name/sweeps/pp7qjzz9 > View run at https://wandb.ai/magenbrot/test-sweep\_project\_name/runs/a0jp1107 > run id: \*\*\_a0jp1107\_\*\* > run name: lilac-sweep-3 > Start KFold 1 > \------------------------- > Start Training Epoch 0 > Start Validation Epoch 0 > \------------------------- > Start Training Epoch 1 > Start Validation Epoch 1 > \------------------------- > Start Training Epoch 2 > Start Validation Epoch 2 > \------------------------- > Start Training Epoch 3 > Start Validation Epoch 3 > Finish KFold 1 > Waiting for W&B process to finish... (success). > 0.067 MB of 0.089 MB uploaded (0.000 MB deduped) > Run history: > > target ▁▃▆█ > val\_target ▁▃▆█ > > Run summary: > > target 3 > val\_target 4 > > View run lilac-sweep-3 at: https://wandb.ai/magenbrot/test-sweep\_project\_name/runs/a0jp1107 > Synced 7 W&B file(s), 0 media file(s), 0 artifact file(s) and 2 other file(s) > Find logs at: .\\wandb\\run-20230309\_121936-a0jp1107\\logs > Waiting for wandb.init()... > Tracking run with wandb version 0.13.11 > Syncing run lilac-sweep-3 to Weights & Biases (docs) > Sweep page: https://wandb.ai/magenbrot/test-sweep\_project\_name/sweeps/pp7qjzz9 > View project at https://wandb.ai/magenbrot/test-sweep\_project\_name > View sweep at https://wandb.ai/magenbrot/test-sweep\_project\_name/sweeps/pp7qjzz9 > View run at https://wandb.ai/magenbrot/test-sweep\_project\_name/runs/a0jp1107 > run id: \*\*\_a0jp1107\_\*\* > run name: lilac-sweep-3 > Start KFold 2 > \------------------------- > Start Training Epoch 0 > Start Validation Epoch 0 > \------------------------- > Start Training Epoch 1 > Start Validation Epoch 1 > \------------------------- > Start Training Epoch 2 > Start Validation Epoch 2 > \------------------------- > Start Training Epoch 3 > Start Validation Epoch 3 > Finish KFold 2 > Waiting for W&B process to finish... (success). > Run history: > > target ▁▃▆█ > val\_target ▁▃▆█ > > Run summary: > > target 3 > val\_target 4 > > View run lilac-sweep-3 at: https://wandb.ai/magenbrot/test-sweep\_project\_name/runs/a0jp1107 > Synced 7 W&B file(s), 0 media file(s), 0 artifact file(s) and 2 other file(s) > Find logs at: .\\wandb\\run-20230309\_121951-a0jp1107\\logs > Waiting for wandb.init()... > Tracking run with wandb version 0.13.11 > Syncing run lilac-sweep-3 to Weights & Biases (docs) > Sweep page: https://wandb.ai/magenbrot/test-sweep\_project\_name/sweeps/pp7qjzz9 > View project at https://wandb.ai/magenbrot/test-sweep\_project\_name > View sweep at https://wandb.ai/magenbrot/test-sweep\_project\_name/sweeps/pp7qjzz9 > View run at https://wandb.ai/magenbrot/test-sweep\_project\_name/runs/a0jp1107 > run id: \*\*\_a0jp1107\_\*\* > run name: lilac-sweep-3 > Start KFold 3 > \------------------------- > Start Training Epoch 0 > Start Validation Epoch 0 > \------------------------- > Start Training Epoch 1 > Start Validation Epoch 1 > \------------------------- > Start Training Epoch 2 > Start Validation Epoch 2 > \------------------------- > Start Training Epoch 3 > Start Validation Epoch 3 > Finish KFold 3 > Waiting for W&B process to finish... (success). > 0.080 MB of 0.090 MB uploaded (0.000 MB deduped) > Run history: > > target ▁▃▆█ > val\_target ▁▃▆█ > > Run summary: > > target 3 > val\_target 4 > > View run lilac-sweep-3 at: https://wandb.ai/magenbrot/test-sweep\_project\_name/runs/a0jp1107 > Synced 7 W&B file(s), 0 media file(s), 0 artifact file(s) and 2 other file(s) > Find logs at: .\\wandb\\run-20230309\_122006-a0jp1107\\logs > > $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ > \*\*\_New Sweep starting here using new group\_\*\* > > wandb: Agent Starting Run: hjtswhd8 with config: > wandb: epochs: 2 > wandb: kfold: 2 > ..... > \`\`\` > > > \### Additional Files > > \_No response\_ > > \### Environment > > WandB version: 0.13.11 (but tested at 0.13.6 with same issue) > > OS: Windows 11 > > Python version: 3.9.16 using mamba > > Versions of relevant libraries: "up to date installed for this python version" > > > > \### Additional Context > > \_No response\_ Should help additionally to reproduce and find error ---
**Author:** ![anmolmann](https://avatars.discourse-cdn.com/v4/letter/a/c37758/32.png) [@anmolmann](https://community.wandb.ai/u/anmolmann)\ **Post date:** [April 10, 2023, 3:04pm UTC](https://community.wandb.ai/t/hyperparameter-tuning-combined-with-k-fold-cross-validation/3881/14 "2023-04-10T15:04:55Z")
Hi @magenbrot , @mbp : I’ve left a reply [here](https://github.com/wandb/wandb/issues/5119#issuecomment-1501879656). Hoping to revive the conversation in the GitHub thread. ---
**Author:** ![system](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/1X/366b649231631dbab896843020da0056074ac79d.png) [@system](https://community.wandb.ai/u/system)\ **Post date:** [June 9, 2023, 3:05pm UTC](https://community.wandb.ai/t/hyperparameter-tuning-combined-with-k-fold-cross-validation/3881/15 "2023-06-09T15:05:22Z")
Closed on June 9, 2023, 3:05pm UTC This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.