100% offline sweep

W&B Help
,
1

Hi,

I’m using wandb sweeps for bayesian optimisation . However I recently got access to new powerfull gpus which are not connected to internet. So i’m trying to use the offline mode of wandb that seems to works fine with regular runs but doesn’t with sweeps. Indeed even in offline mode a connexion seems to still be required in order to create the sweep and to synchronise the data after a run (for the optimisation process).

Is there any way to run a 100% offline sweep and to synchronise it manually at the end off each run or of the whole sweep ?

2

Hi Felix,

Unfortunately, there is no way for us to run sweeps in offline mode, this feature is not available as of right now.

If you would like I could create a feature request for it, bump up the priority, and submit it to our engineers.

Warmly,
Artsiom

2 Likes
3

Hi Félix,

We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.

Best,
Weights & Biases

4

Hi Félix,
Since we have not heard back from you, we will close this request. If you would like to re-open the conversation, please let us know!

Cheers!

Weights & Biases

5

Hi,

I also run wandb on an Slurm cluster in which computational nodes do not have internet connection. I am really interested in this feature. Has the feature request been created?

1 Like
6

Same here! I really need that feature

1 Like
7

I really need this feature too! @artsiom , are there any workarounds?

1 Like
8

@artsiom , to clarify my use case, I can create a sweep just fine (i.e. wandb sweep <path to YAML config> works). Previous, I would launch SLURM jobs, each with 1 W&B agent, but due to my SLURM cluster’s new policies, each SLURM job has no internet access and thus no agent can acquire a run.

Is there a workaround?

9

I would love this feature too! To be honest I wouldn’t know how one could manage the run configuration exchange with the agent without connection, but running sweeps offline would be great!

1 Like
10

Same here. Unfortunately most slurm cluster do not support network connection.
Hope update soon.

1 Like
11

I now have this problem with 2 other clusters. Can someone from W&B please suggest a solution, or discuss a workaround? I’m wondering if we can somehow persuade our clusters to run a W&B server internally that then gets occasionally synced to W&B’s machines?

@artsiom , could you perhaps advise?

1 Like
12

@artsiom I share this interest just like the other users, but I have not seen a reply in a long time. Do we need to make a new post?

13

Hi all,
I am woundering if this feature has been implemented yet ?
I also need to use it since most of the clusters I have access to has no internet access.
Best regards

14

Hi Guys, for some reason haven’t been getting notifications for this thread at all! Apologies to everyone. I have followed up with the our eng department and still waiting for a reply. I’ll make another feature request for you. But I would also advise making a new ticket, for visibility to show how many people want this feature request.

15

As a workaround,

You can now potentially use the Local Controller, then when ready, you can sync your runs to your profile.

1 Like
16

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.