Setting up a sweep using port forwarding?

W&B Help
1

I have a compute environment in which I have a university hosted cluster (so I am not admin) with a login node connected to the internet and not meant for compute jobs and several compute nodes which are connected to the login node but not to the internet. I would love to run a sweep in this environment but in the standard setup I would need my compute nodes to be connected to the internet. Is there a way I can setup port forwarding on the compute nodes so they can access the W&B server via the login node? For reference, all environments are Ubuntu 18.04.6

2

Hi Evan,

Currently we don’t have a way to have a completely offline version of sweeps. But a workaround that we do have is to run wandb local and use wandb offline, then use wandb sync to sync those results to your computer that has access to the internet.

2 Likes
3

Hey @lesliewandb ,
Thanks so much for your help. That is the workaround I am using right now although it is a bit cumbersome. Is there no way to engineer a way for “offline” nodes to communicate with the wandb server through an online node they are connected to? I definitely understand that a fully offline node could not but I’d imagine if ports are aligned correctly it could be done. Do you know if there is a section of the wandb sweep sdk that deals with this which I could explore?

4

I’m sorry, but right now that’s the only option we have for offline sweeps

5

I assume your compute nodes have ssh access to the login node. If that’s the case, have you tried to set up a socks proxy using a ssh tunnel? You can follow any online tutorial, like this one: Create a SOCKS proxy on a Linux server with SSH to bypass content filters.

After you have set up your tunnel, you could set the following environment variables so your proxy is picked up by the requests library for http and https traffic:

export HTTP_PROXY="socks5://localhost:1337"
export HTTPS_PROXY="socks5://localhost:1337"

Assuming wandb uses requests and http or https traffic (which I don’t really know), this should work.

6

Thanks so much @pcuenq ! I am speaking with the sysadmins at my server about setting this up

7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.