Unable to run wandb online after running offline

I have a compute environment where I was running wandb offline for quite a while. I am now hoping to use it online (to get automatic syncing), however I seem to be unable to set this up now. The following is a minimal reproducible example:

>> import wandb
>> test = wandb.init(mode='online')
Traceback (most recent call last):
  File "[path]/lib/python3.9/site-packages/wandb/sdk/wandb_init.py", line 867, in init
    wi.setup(kwargs)
  File "[path]/lib/python3.9/site-packages/wandb/sdk/wandb_init.py", line 182, in setup
    user_settings = self._wl._load_user_settings()
  File "[path]/lib/python3.9/site-packages/wandb/sdk/wandb_setup.py", line 183, in _load_user_settings
    flags = self._server._flags
AttributeError: 'NoneType' object has no attribute '_flags'
wandb: ERROR Abnormal program exit

I have tried

  • running wandb online in the terminal
  • setting the wandb mode environment variable to be online
  • uninstalling and reinstalling wandb

Is there any way I can run this online?

Hi @evanv, you shouldn’t have a problem changing between online and offline, can you give me the version of wandb you’re using?

Thanks so much! I am using wandb v 0.12.6. In case it is useful I also wanted to make you aware of a peculiarity of the system I am using. It is a shared HPC and if I try to use wandb in a folder which does not have its own wandb folder, I get the following:

(my_conda_env) user@login-2:[path to folder with no wandb folder]$ wandb --version
Traceback (most recent call last):
  File "[conda env path]/bin/wandb", line 5, in <module>
    from wandb.cli.cli import cli
  File "[conda env path]/lib/python3.9/site-packages/wandb/cli/cli.py", line 48, in <module>
    logging.basicConfig(
  File "[conda env path]/lib/python3.9/logging/__init__.py", line 2003, in basicConfig
    h = FileHandler(filename, mode,
  File "[conda env path]/lib/python3.9/logging/__init__.py", line 1146, in __init__
    StreamHandler.__init__(self, self._open())
  File "[conda env path]/lib/python3.9/logging/__init__.py", line 1175, in _open
    return open(self.baseFilename, self.mode, encoding=self.encoding,
PermissionError: [Errno 13] Permission denied: '/tmp/debug-cli.log'
(my_conda_env) user@login-2:[path to folder with no wandb folder]$ cd [folder with wandb folder]
(my_conda_env) user@login-2:[path to folder with wandb folder]$ wandb --version
wandb, version 0.12.6
1 Like

@evanv It looks like your /tmp directory is not writable. if you run ls -al /tmp in your terminal I’m guessing we will see that this directory is not readable except by root users (hence why you need to run sudo to read from the directory). You should be able to fix this by issuing the command chmod +r /tmp from your terminal. Or, if you can’t change the permissions, the my recommendation would be to specify a directory by specific the env var: WANDB_DIR=/home/username/tmp.
Hope this helps.

Hi @anmolmann , thank you for letting me know! Indeed /tmp is not writable. I tried modifying the WANDB_DIR environment variable, both in my .bash_profile and just in the terminal, to an existing and writeable directory `/home/my_username/tmp/’. Unfortunately, the above problem still persists :frowning:

Hey @evanv, we should have a fix in for the original issue posted at the top here, however I suspect that it might not resolve your problem. I have a couple questions for you:

  • You mentioned that you were using the offline mode – do you know where things were logged to locally? wandb should have said something like
wandb: W&B syncing is set to `offline` in this directory.  
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.

wandb: Waiting for W&B process to finish, PID 85946... (success).
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /<path>/wandb/offline-run-<tag>-3ktlmbnl
wandb: Find logs at: ./wandb/offline-run-<tag>-3ktlmbnl/logs/debug.log
  • In your script where you init wandb - what does print(os.environ.get("WANDB_DIR")) print, the expected value you set (/home/your_username/tmp/)? Have you tried setting it inside the script itself with os.environ["WANDB_DIR"] = "/home/your_username/tmp/"?

Thanks!

Also, @evanv, have you tried initializing wandb like this:

wandb.init(dir="/home/your_username/tmp/")

?

Hey @dimaduev , thanks so much for the fixes! I think I was able to resolve this through looking at the different WANDB_DIR locations… I had several in different bashrc/zshrc files and I suspect this was causing an issue. It seems to be resolved now!

1 Like

Excellent, glad to hear that!