# Hypyerparameter optimization with k folds on each iteration

**URL:** <https://community.wandb.ai/t/hypyerparameter-optimization-with-k-folds-on-each-iteration/3429>\
**Category:** W&B Help\
**Tags:** sweeps\
**Created:** [November 16, 2022, 3:55pm UTC](https://community.wandb.ai/t/hypyerparameter-optimization-with-k-folds-on-each-iteration/3429 "2022-11-16T15:55:13Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![aulrichsen](https://avatars.discourse-cdn.com/v4/letter/a/e8c25b/32.png) [@aulrichsen](https://community.wandb.ai/u/aulrichsen)\
**Post date:** [November 16, 2022, 3:55pm UTC](https://community.wandb.ai/t/hypyerparameter-optimization-with-k-folds-on-each-iteration/3429/1 "2022-11-16T15:55:13Z")

</div>

I am trying to perform hyperparameter optimization with wandb and for each iteration I would like to get the average performance across 3 different folds of my dataset.

I have defined a function optimize that i pass to wandb.agent:

```auto
def optimize(config):
    for fold in range(1, 4):    
        dataset_artifact = f'fold-{fold}:latest'
        config['dataset_artifact'] = dataset_artifact 
        with wandb.init(config=config, group=group_name, job_type=f'train-fold-{fold}', name=f'train-fold-{fold}', reinit=True) as run:   
            train_and_log(config, run)  
            run.finish()

```

I would expect this to creat a seperate run for each fold (since I have specified a different job type and run name as well as passing init=True) so that I would end up with:

Group: param\_combo\_1

&nbsp;&nbsp;&nbsp;&nbsp;\> Job Type: train-fold-1

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\> train-fold-1

&nbsp;&nbsp;&nbsp;&nbsp;\> Job Type: train-fold-2

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\> train-fold-2

&nbsp;&nbsp;&nbsp;&nbsp;\> Job Type: train-fold-3

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\> train-fold-3

However each run for a given hyperparameter iteration overwrites the previous fold so I in fact end up with

Group: param\_combo\_1

&nbsp;&nbsp;&nbsp;&nbsp;\> Job Type: train-fold-3

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\> train-fold-3

How can I resolve this issue?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/1X/366b649231631dbab896843020da0056074ac79d.png) [@system](https://community.wandb.ai/u/system)\
**Post date:** [November 17, 2022, 3:09pm UTC](https://community.wandb.ai/t/hypyerparameter-optimization-with-k-folds-on-each-iteration/3429/2 "2022-11-17T15:09:37Z")

</div>

Hi Alexander, thanks for writing in!

I can see the same behaviour as you do. After some investigation, I have realized that this is an intended one because of the fact that, with each combination of parameters it is created only one run (same run id), so it is only resuming the previous run although you use `reinit=True`. In terms of a workaround, I think there are two ways to solve this:

- Average your metrics inside the `optimize()`/`train_and_log()` function in the same run instead of creating different runs.
- Use the `grid` method instead of `random` and repeat some values (i.e. batch\_size=[64,64,64,128,128,128]).  
Please let me know if any of these would work for you or if you would like me to create a request for this feature (I was thinking something like a new argument in the agent like `repeat=number_of_repetitions` and average the results). If this is the case, I would really appreciate if you could give me some more details about your use-case and why this new feature would be useful for you. Thanks!

Best,  
Luis

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/1X/366b649231631dbab896843020da0056074ac79d.png) [@system](https://community.wandb.ai/u/system)\
**Post date:** [November 22, 2022, 10:28am UTC](https://community.wandb.ai/t/hypyerparameter-optimization-with-k-folds-on-each-iteration/3429/3 "2022-11-22T10:28:46Z")

</div>

Hi Alexander,

We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.

Best,  
Luis

---

<div class="post-metadata">

**Author:** ![aulrichsen](https://avatars.discourse-cdn.com/v4/letter/a/e8c25b/32.png) [@aulrichsen](https://community.wandb.ai/u/aulrichsen)\
**Post date:** [November 25, 2022, 11:57am UTC](https://community.wandb.ai/t/hypyerparameter-optimization-with-k-folds-on-each-iteration/3429/4 "2022-11-25T11:57:33Z")

</div>

Hi Luis,

Those workarounds will probably be okay but I would very much like to see a feature allowing you to implement this behaviour with a parameter such as ‘repeat’ as you mentioned.

Many thanks,  
Alexander

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/1X/366b649231631dbab896843020da0056074ac79d.png) [@system](https://community.wandb.ai/u/system)\
**Post date:** [November 28, 2022, 10:49am UTC](https://community.wandb.ai/t/hypyerparameter-optimization-with-k-folds-on-each-iteration/3429/5 "2022-11-28T10:49:07Z")

</div>

Hi Alexander,

Thanks for confirming this! I have created a request for this feature, thanks for suggesting it. May I help you in any other way?

Best,  
Luis

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/1X/366b649231631dbab896843020da0056074ac79d.png) [@system](https://community.wandb.ai/u/system)\
**Post date:** [January 24, 2023, 11:58am UTC](https://community.wandb.ai/t/hypyerparameter-optimization-with-k-folds-on-each-iteration/3429/6 "2023-01-24T11:58:07Z")

</div>

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.