I’m using wandb sweeps for bayesian optimisation . However I recently got access to new powerfull gpus which are not connected to internet. So i’m trying to use the offline mode of wandb that seems to works fine with regular runs but doesn’t with sweeps. Indeed even in offline mode a connexion seems to still be required in order to create the sweep and to synchronise the data after a run (for the optimisation process).
Is there any way to run a 100% offline sweep and to synchronise it manually at the end off each run or of the whole sweep ?
We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.
I also run wandb on an Slurm cluster in which computational nodes do not have internet connection. I am really interested in this feature. Has the feature request been created?
@artsiom , to clarify my use case, I can create a sweep just fine (i.e. wandb sweep <path to YAML config> works). Previous, I would launch SLURM jobs, each with 1 W&B agent, but due to my SLURM cluster’s new policies, each SLURM job has no internet access and thus no agent can acquire a run.
I would love this feature too! To be honest I wouldn’t know how one could manage the run configuration exchange with the agent without connection, but running sweeps offline would be great!