# Repeated wandb.init() in parallelized wandb sweeps

**URL:** <https://community.wandb.ai/t/repeated-wandb-init-in-parallelized-wandb-sweeps/8501>\
**Category:** W&B Help\
**Tags:** sweeps, wandb\
**Created:** [November 30, 2024, 3:18pm UTC](https://community.wandb.ai/t/repeated-wandb-init-in-parallelized-wandb-sweeps/8501 "2024-11-30T15:18:24Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![leofierus](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/leofierus/32/2241_2.png) [@leofierus](https://community.wandb.ai/u/leofierus)\
**Post date:** [November 30, 2024, 3:18pm UTC](https://community.wandb.ai/t/repeated-wandb-init-in-parallelized-wandb-sweeps/8501/1 "2024-11-30T15:18:24Z")

</div>

Hi wandb community. I wrote some code trying to parallelize my wandb sweeps since the model I am working with takes a long time to converge and I have a lot of subprocesses to sweep through. Basically I don’t have the luxury of time right now. Here’s a generalized snippet of my code:

```python
def run_pipeline(args):
    # Stuff happens here

    # Wandb init
    group = "within_session" if session_config["within_session"] else "across_session"
    run = wandb.init(name=f"{sessions[i]}_{group}_decoder_run", group=group, config=sweep_config, reinit=True)

    # Model training

    return results

def run_pipeline_wrapper(args):
    # Stuff happens here
    run_pipeline(args)

    return None

if __name__ == " __main__":
    total_runs = 30
    agents = 5
    runs_per_agent = total_runs // agents

    sweep_config = {'method': 'random'}
    parameters_dict = {
        # Lota of parameters to sweep
    }
    sweep_config['parameters'] = parameters_dict

    # Create a sweep id that stores sweep ids
    sweep_id_json_path = 'sweep_id.json'
    if not os.path.exists(sweep_id_json_path):
        with open(sweep_id_json_path, 'w') as f:
            json.dump({}, f)
    sweep_id_json = json.load(open(sweep_id_json_path, 'r'))

    # Sessions_list = number of unique data that I need to run my sweeps
    for i in range(len(sessions_list)):

        # Preparing a partial method to pass
        run_pipeline_with_args = partial(run_pipeline_wrapper, args)

        # I cache the existing sweep_ids in a json file to help in attaching sweep ids if I rerun the code again
        if f"{sessions_list[i]}_{is_within}" not in sweep_id_json:
            sweep_id = wandb.sweep(sweep_config, project=f"HPC_model_{sess}_session_{data}_{data_type}")
        else:
            sweep_id = wandb.sweep(sweep_config, project=f"HPC_model_{sess}_session_{data}_{data_type}"
                                   , prior_runs=sweep_id_json[f"{sessions_list[i]}_{is_within}"])

        # This is the parallelization logic, where I parallelize the sweeps
        with concurrent.futures.ThreadPoolExecutor(max_workers=agents) as executor:
            futures = [
                executor.submit(wandb.agent, sweep_id, run_pipeline_with_args, count=runs_per_agent)
                for _ in range(agents)
            ]

            concurrent.futures.wait(futures)

```

When I run this code, I am basically stuck on wandb.init(), with that process eventually being terminated due to a timeout. I don’t think this is a problem of increasing wandb’s timeout. How do I fix this? Do you think this might be a problem because of my parallelization logic? If so, how do you devs parallelize your wandb sweeps in-code?

Attached logs:

 ![image](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/2X/5/56f3450d67215a68911ce0ec245b5d2f070af523.png)