Wandb online/offline status breaks when using WANDB_DIR environment variable

Hi, while starting to get familiar with wandb I ran into a bug.

I was testing a very simple setup with the following file:

train.py

import wandb
import random
import logging

log = logging.getLogger(__name__)


def train():
    with wandb.init(
        entity="jeroenboss",
        project="wandb_demo"
    ) as run:
        log.info(f"Running experiment with name {run.name}")
        config = run.config

        log.info(f"Loaded configuration: {config}")
        accuracy = 0
        for x in range(config.get('epochs', 100)):
            accuracy += (1 - accuracy) * (random.random()*0.9+0.1) * config.get('alpha', 0.1)

            run.log({"accuracy": accuracy})

        log.info("Finished")


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    train()

Because I was testing a few things I wanted to prevent uploading the runs to your servers before I had figured out some bugs. But when I tried to use the wandb offline command, it did not seem to work.
After some testing I found that setting WANDB_DIR in my ~/.bashrc breaks the functionality. Below you can see the output of my commands. Removing WANDB_DIR made everything work correctly.

(wandb_demo) jeroen@jeroen-ThinkPad-P50:~/sandbox/wandb_test$ wandb status
Current Settings
{
  "base_url": "https://api.wandb.ai",
  "entity": null,
  "git_remote": "origin",
  "ignore_globs": [],
  "project": null,
  "section": "default"
}
(wandb_demo) jeroen@jeroen-ThinkPad-P50:~/sandbox/wandb_test$ wandb offline
W&B offline, running your script from this directory will only write metadata locally.
(wandb_demo) jeroen@jeroen-ThinkPad-P50:~/sandbox/wandb_test$ wandb status
Current Settings
{
  "base_url": "https://api.wandb.ai",
  "disabled": "true",
  "entity": null,
  "git_remote": "origin",
  "ignore_globs": [],
  "mode": "offline",
  "project": null,
  "section": "default"
}
(wandb_demo) jeroen@jeroen-ThinkPad-P50:~/sandbox/wandb_test$ python train.py
wandb: Currently logged in as: jeroenboss (use `wandb login --relogin` to force relogin)
wandb: Tracking run with wandb version 0.12.6
wandb: Syncing run misunderstood-night-24
wandb: ⭐️ View project at https://wandb.ai/jeroenboss/wandb_demo
wandb: πŸš€ View run at https://wandb.ai/jeroenboss/wandb_demo/runs/kv7r7fst
wandb: Run data is saved locally in /home/jeroen/.wandb/runs/wandb/run-20211105_121404-kv7r7fst
wandb: Run `wandb offline` to turn off syncing.

INFO:__main__:Running experiment with name misunderstood-night-24
INFO:__main__:Loaded configuration: {'epochs': 50, 'alpha': 0.1}
INFO:__main__:Finished

wandb: Waiting for W&B process to finish, PID 17238... (success).
wandb:                                                                                
wandb: Run history:
wandb:   accuracy β–β–‚β–‚β–‚β–ƒβ–ƒβ–ƒβ–„β–„β–„β–…β–…β–…β–…β–…β–†β–†β–†β–†β–†β–‡β–‡β–‡β–‡β–‡β–‡β–‡β–‡β–‡β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ
wandb: 
wandb: Run summary:
wandb:   accuracy 0.95235
wandb: 
wandb: Synced 7 W&B file(s), 0 media file(s), 0 artifact file(s) and 1 other file(s)
wandb: Synced misunderstood-night-24: https://wandb.ai/jeroenboss/wandb_demo/runs/kv7r7fst
wandb: Find logs at: /home/jeroen/.wandb/runs/wandb/run-20211105_121404-kv7r7fst/logs/debug.log
wandb: 
(wandb_demo) jeroen@jeroen-ThinkPad-P50:~/sandbox/wandb_test$ wandb status
Current Settings
{
  "base_url": "https://api.wandb.ai",
  "disabled": "true",
  "entity": null,
  "git_remote": "origin",
  "ignore_globs": [],
  "mode": "offline",
  "project": null,
  "section": "default"
}

I’m running Python 3.7.12 and wandb 0.12.6

I would like to store my runs in a central folder, so I would like to be able to use WANDB_DIR.

Hi Jeroen,

Can you show me what errors would pop up before you removed WANDB_DIR?

Warmly,
Leslie

What do you mean by that?

If I don’t set the WANDB_DIR environment variable then there are no errors and the wandb status is working as expected.
If I do set the WANDB_DIR environment variable then there are also no errors but the wandb status does not work as expected.

It seems like wandb does not check the correct location to retrieve the mode that is active.

This problem still persists. Could someone have a look?

The problem is easily replicated:

  1. set the WANDB_DIR environment variable
  2. run the wandb offline command
  3. run a python script to run wandb.init

The init call will create a run in online mode, even though it is storing the results in the WANDB_DIR. The settings file in that directory is ignored.

Is your WANDB_DIR variable set to ? Is this a HPC cluster, your own machine, or somewhere else? What permissions do you have for your device.

Hi Jeroen,

We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.

Best,
Weights & Biases

Hi Jeroen,
Since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!