I have a compute environment in which I have a university hosted cluster (so I am not admin) with a login node connected to the internet and not meant for compute jobs and several compute nodes which are connected to the login node but not to the internet. I would love to run a sweep in this environment but in the standard setup I would need my compute nodes to be connected to the internet. Is there a way I can setup port forwarding on the compute nodes so they can access the W&B server via the login node? For reference, all environments are Ubuntu 18.04.6
Currently we don’t have a way to have a completely offline version of sweeps. But a workaround that we do have is to run wandb local and use wandb offline, then use wandb sync to sync those results to your computer that has access to the internet.
Hey @lesliewandb ,
Thanks so much for your help. That is the workaround I am using right now although it is a bit cumbersome. Is there no way to engineer a way for “offline” nodes to communicate with the wandb server through an online node they are connected to? I definitely understand that a fully offline node could not but I’d imagine if ports are aligned correctly it could be done. Do you know if there is a section of the wandb sweep sdk that deals with this which I could explore?
I’m sorry, but right now that’s the only option we have for offline sweeps