Run self-hosted Wandb server on different machine than training jobs

Hi Wandb,

I was wondering if the following is possible:

  1. Run wandb-tracked experiments on an external machine without internet access
  2. Send the folder containing all wandb logs, meta-data, etc. to my local machine
  3. Run self-hosted wandb server on my local machine and display the experiment tracking

An important constraint is that my project doesn’t allow to store any data externally. I know this set-up is possible with Tensorboard, but I would preferably use Wandb :slight_smile:

Thanks!

Hi @adrisk_intelligence!

There are a few ways you can do this -

  1. If you are running a local instance on another machine and it is connected to a network, you can log in to your instance through wandb login to stream data directly to it and use W&B as normal
wandb login --host=http://link.to.your/instance:PORT
  1. If your instance is not connected to a network, there is also a way to go about it, though it is a little more long winded:
    1. Set your machine which is going to be training to offline mode through wandb offline or wandb.init(..., mode='offline') (this is done so that your training machine does not try to stream logs anywhere)
    2. After training, notice that there is a folder called wandb in your working directory.
    3. Move this directory to your machine hosting your instance, this is going to contain all your run information. This can be done through scp or any other method you are comfortable with.
    4. Lastly, you will want to run wandb sync to synchronize your runs to the server.

Please note there are a few things to keep in mind when you are using the second method, listed below:

  • When using wandb sync, we create a .synced file for each run to make sure that the run has been synced. Do not delete those files, otherwise those runs will attempt to sync again, which has a chance to cause an SQL Collision in our instance. Specifically, there is a chance that these files get overwritten when you communicate the wandb folder manually across machines, which is why I wanted to point this out.
  • On a similar note, you probably want to use the --no-include-synced flag with wandb sync to be sure that your runs do not get synced multiple times. You can read more about the flags we allow on our documentation here.

Thanks,
Ramit

Hi Christoffel,

We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.

Best,
Weights & Biases