I have a compute environment where I was running wandb offline for quite a while. I am now hoping to use it online (to get automatic syncing), however I seem to be unable to set this up now. The following is a minimal reproducible example:
>> import wandb
>> test = wandb.init(mode='online')
Traceback (most recent call last):
File "[path]/lib/python3.9/site-packages/wandb/sdk/wandb_init.py", line 867, in init
wi.setup(kwargs)
File "[path]/lib/python3.9/site-packages/wandb/sdk/wandb_init.py", line 182, in setup
user_settings = self._wl._load_user_settings()
File "[path]/lib/python3.9/site-packages/wandb/sdk/wandb_setup.py", line 183, in _load_user_settings
flags = self._server._flags
AttributeError: 'NoneType' object has no attribute '_flags'
wandb: ERROR Abnormal program exit
I have tried
running wandb online in the terminal
setting the wandb mode environment variable to be online
Thanks so much! I am using wandb v 0.12.6. In case it is useful I also wanted to make you aware of a peculiarity of the system I am using. It is a shared HPC and if I try to use wandb in a folder which does not have its own wandb folder, I get the following:
(my_conda_env) user@login-2:[path to folder with no wandb folder]$ wandb --version
Traceback (most recent call last):
File "[conda env path]/bin/wandb", line 5, in <module>
from wandb.cli.cli import cli
File "[conda env path]/lib/python3.9/site-packages/wandb/cli/cli.py", line 48, in <module>
logging.basicConfig(
File "[conda env path]/lib/python3.9/logging/__init__.py", line 2003, in basicConfig
h = FileHandler(filename, mode,
File "[conda env path]/lib/python3.9/logging/__init__.py", line 1146, in __init__
StreamHandler.__init__(self, self._open())
File "[conda env path]/lib/python3.9/logging/__init__.py", line 1175, in _open
return open(self.baseFilename, self.mode, encoding=self.encoding,
PermissionError: [Errno 13] Permission denied: '/tmp/debug-cli.log'
(my_conda_env) user@login-2:[path to folder with no wandb folder]$ cd [folder with wandb folder]
(my_conda_env) user@login-2:[path to folder with wandb folder]$ wandb --version
wandb, version 0.12.6
@evanv It looks like your /tmp directory is not writable. if you run ls -al /tmp in your terminal I’m guessing we will see that this directory is not readable except by root users (hence why you need to run sudo to read from the directory). You should be able to fix this by issuing the command chmod +r /tmp from your terminal. Or, if you can’t change the permissions, the my recommendation would be to specify a directory by specific the env var: WANDB_DIR=/home/username/tmp.
Hope this helps.
Hi @anmolmann , thank you for letting me know! Indeed /tmp is not writable. I tried modifying the WANDB_DIR environment variable, both in my .bash_profile and just in the terminal, to an existing and writeable directory `/home/my_username/tmp/’. Unfortunately, the above problem still persists
Hey @evanv, we should have a fix in for the original issue posted at the top here, however I suspect that it might not resolve your problem. I have a couple questions for you:
You mentioned that you were using the offline mode – do you know where things were logged to locally? wandb should have said something like
wandb: W&B syncing is set to `offline` in this directory.
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
wandb: Waiting for W&B process to finish, PID 85946... (success).
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /<path>/wandb/offline-run-<tag>-3ktlmbnl
wandb: Find logs at: ./wandb/offline-run-<tag>-3ktlmbnl/logs/debug.log
In your script where you init wandb - what does print(os.environ.get("WANDB_DIR")) print, the expected value you set (/home/your_username/tmp/)? Have you tried setting it inside the script itself with os.environ["WANDB_DIR"] = "/home/your_username/tmp/"?
Hey @dimaduev , thanks so much for the fixes! I think I was able to resolve this through looking at the different WANDB_DIR locations… I had several in different bashrc/zshrc files and I suspect this was causing an issue. It seems to be resolved now!
