100% offline sweep

Hi,

I’m using wandb sweeps for bayesian optimisation . However I recently got access to new powerfull gpus which are not connected to internet. So i’m trying to use the offline mode of wandb that seems to works fine with regular runs but doesn’t with sweeps. Indeed even in offline mode a connexion seems to still be required in order to create the sweep and to synchronise the data after a run (for the optimisation process).

Is there any way to run a 100% offline sweep and to synchronise it manually at the end off each run or of the whole sweep ?

Hi Felix,

Unfortunately, there is no way for us to run sweeps in offline mode, this feature is not available as of right now.

If you would like I could create a feature request for it, bump up the priority, and submit it to our engineers.

Warmly,
Artsiom

2 Likes

Hi Félix,

We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.

Best,
Weights & Biases

Hi Félix,
Since we have not heard back from you, we will close this request. If you would like to re-open the conversation, please let us know!

Cheers!

Weights & Biases

Hi,

I also run wandb on an Slurm cluster in which computational nodes do not have internet connection. I am really interested in this feature. Has the feature request been created?

1 Like

Same here! I really need that feature

1 Like

I really need this feature too! @artsiom , are there any workarounds?

1 Like

@artsiom , to clarify my use case, I can create a sweep just fine (i.e. wandb sweep <path to YAML config> works). Previous, I would launch SLURM jobs, each with 1 W&B agent, but due to my SLURM cluster’s new policies, each SLURM job has no internet access and thus no agent can acquire a run.

Is there a workaround?

I would love this feature too! To be honest I wouldn’t know how one could manage the run configuration exchange with the agent without connection, but running sweeps offline would be great!

Same here. Unfortunately most slurm cluster do not support network connection.
Hope update soon.