ServiceStartProcessError in Jupyter Notebook

W&B Help
,
1

Hi,
I am getting the following error when running wandb.init() in a jupyter notebook:

Some other info:
print(sys.executable) >> /da* ta/miniconda3/envs/prefopt1/bin/python

  • wandb.version >> 0.16.3 (I have also tried 0.15.0 as suggested in another thread but I get the same error.
  • Wandb works fine from a normal python script - the error is when I try using it in a jupyter notebook.
  • I tried pip install -U click as suggested here Unable to login - #12 by andravin, but this did not change anything.

Any suggestions would be really appreciated!

Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb: WARNING Path /data/ziz/not-backed-up/scratch/sims/rl-from-feedback/d2po-outer/d2po-results/wandb_out/wandb/ wasn't writable, using system temp directory
Traceback (most recent call last):
  File "/data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/site-packages/wandb/__main__.py", line 1, in <module>
    from wandb.cli import cli
  File "/data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/site-packages/wandb/cli/cli.py", line 61, in <module>
    logging.basicConfig(
  File "/data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/logging/__init__.py", line 1988, in basicConfig
    h = FileHandler(filename, mode)
  File "/data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/logging/__init__.py", line 1147, in __init__
    StreamHandler.__init__(self, self._open())
  File "/data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/logging/__init__.py", line 1176, in _open
    return open(self.baseFilename, self.mode, encoding=self.encoding)
PermissionError: [Errno 13] Permission denied: '/tmp/wandb/debug-cli.sims.log'
---------------------------------------------------------------------------
ServiceStartProcessError                  Traceback (most recent call last)
Cell In[3], line 1
----> 1 wandb.init()

File /data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/site-packages/wandb/sdk/wandb_init.py:1164, in init(job_type, dir, config, project, entity, reinit, tags, group, name, notes, magic, config_exclude_keys, config_include_keys, anonymous, mode, allow_val_change, resume, force, tensorboard, sync_tensorboard, monitor_gym, save_code, id, settings)
   1162     if logger is not None:
   1163         logger.exception(str(e))
-> 1164     raise e
   1165 except KeyboardInterrupt as e:
   1166     assert logger

File /data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/site-packages/wandb/sdk/wandb_init.py:1141, in init(job_type, dir, config, project, entity, reinit, tags, group, name, notes, magic, config_exclude_keys, config_include_keys, anonymous, mode, allow_val_change, resume, force, tensorboard, sync_tensorboard, monitor_gym, save_code, id, settings)
   1139 try:
   1140     wi = _WandbInit()
-> 1141     wi.setup(kwargs)
   1142     assert wi.settings
   1143     except_exit = wi.settings._except_exit

File /data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/site-packages/wandb/sdk/wandb_init.py:172, in _WandbInit.setup(self, kwargs)
    169 _disable_service = mode == "disabled" or settings_mode == "disabled"
    170 setup_settings = {"_disable_service": _disable_service}
--> 172 self._wl = wandb_setup.setup(settings=setup_settings)
    173 # Make sure we have a logger setup (might be an early logger)
    174 assert self._wl is not None

File /data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/site-packages/wandb/sdk/wandb_setup.py:327, in setup(settings)
    324 def setup(
    325     settings: Optional[Settings] = None,
    326 ) -> Optional["_WandbSetup"]:
--> 327     ret = _setup(settings=settings)
    328     return ret

File /data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/site-packages/wandb/sdk/wandb_setup.py:320, in _setup(settings, _reset)
    318     _WandbSetup._instance = None
    319     return None
--> 320 wl = _WandbSetup(settings=settings)
    321 return wl

File /data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/site-packages/wandb/sdk/wandb_setup.py:303, in _WandbSetup.__init__(self, settings)
    301     _WandbSetup._instance._update(settings=settings)
    302     return
--> 303 _WandbSetup._instance = _WandbSetup__WandbSetup(settings=settings, pid=pid)

File /data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/site-packages/wandb/sdk/wandb_setup.py:114, in _WandbSetup__WandbSetup.__init__(self, pid, settings, environ)
    111 wandb.termsetup(self._settings, logger)
    113 self._check()
--> 114 self._setup()
    116 tracelog_mode = self._settings._tracelog
    117 if tracelog_mode:

File /data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/site-packages/wandb/sdk/wandb_setup.py:250, in _WandbSetup__WandbSetup._setup(self)
    249 def _setup(self) -> None:
--> 250     self._setup_manager()
    252     sweep_path = self._settings.sweep_param_path
    253     if sweep_path:

File /data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/site-packages/wandb/sdk/wandb_setup.py:277, in _WandbSetup__WandbSetup._setup_manager(self)
    275 if self._settings._disable_service:
    276     return
--> 277 self._manager = wandb_manager._Manager(settings=self._settings)

File /data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/site-packages/wandb/sdk/wandb_manager.py:145, in _Manager.__init__(self, settings)
    143 token = _ManagerToken.from_environment()
    144 if not token:
--> 145     self._service.start()
    146     host = "localhost"
    147     if use_grpc:

File /data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/site-packages/wandb/sdk/service/service.py:199, in _Service.start(self)
    198 def start(self) -> None:
--> 199     self._launch_server()

File /data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/site-packages/wandb/sdk/service/service.py:193, in _Service._launch_server(self)
    191     self._wait_for_ports(fname, proc=internal_proc)
    192 except Exception as e:
--> 193     _sentry.reraise(e)
    194 self._startup_debug_print("wait_ports_done")
    195 self._internal_proc = internal_proc

File /data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/site-packages/wandb/analytics/sentry.py:146, in Sentry.reraise(self, exc)
    143 self.exception(exc)
    144 # this will messily add this "reraise" function to the stack trace,
    145 # but hopefully it's not too bad
--> 146 raise exc.with_traceback(sys.exc_info()[2])

File /data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/site-packages/wandb/sdk/service/service.py:191, in _Service._launch_server(self)
    189 self._startup_debug_print("wait_ports")
    190 try:
--> 191     self._wait_for_ports(fname, proc=internal_proc)
    192 except Exception as e:
    193     _sentry.reraise(e)

File /data/ziz/not-backed-up/sims/miniconda3/envs/test/lib/python3.8/site-packages/wandb/sdk/service/service.py:116, in _Service._wait_for_ports(self, fname, proc)
    101 if proc and proc.poll():
    102     # process finished
    103     # define these variables for sentry context grab:
   (...)
    107     # proc_out = proc.stdout.read()
    108     # proc_err = proc.stderr.read()
    109     context = dict(
    110         command=proc.args,
    111         sys_executable=sys.executable,
   (...)
    114         proc_err=proc.stderr.read() if proc.stderr else "",
    115     )
--> 116     raise ServiceStartProcessError(
    117         f"The wandb service process exited with {proc.returncode}. "
    118         "Ensure that `sys.executable` is a valid python interpreter. "
    119         "You can override it with the `_executable` setting "
    120         "or with the `WANDB__EXECUTABLE` environment variable.",
    121         context=context,
    122     )
    123 if not os.path.isfile(fname):
    124     time.sleep(0.2)

ServiceStartProcessError: The wandb service process exited with 1. Ensure that `sys.executable` is a valid python interpreter. You can override it with the `_executable` setting or with the `WANDB__EXECUTABLE` environment variable.
2

hey @anyasims - Few questions to dig into this:

The error you’re encountering when running wandb.init() in a Jupyter notebook, despite having success with a normal Python script, suggests there might be an environment or permissions issue specific to the Jupyter notebook context. Here are a few steps and considerations that might help resolve the issue:

  1. Permissions Issue: The error message indicates a permission denied error when trying to write to /tmp/wandb/debug-cli.sims.log. This suggests a permissions issue with the ability to write to tmp/wandb/. Are you able to write to the tmp folder?
  • If the issue is related to writing to the temporary directory, you can also try setting the WANDB_DIR environment variable to a path where you have write permissions:
  1. Python Executable: The error mentions ensuring that sys.executable is a valid Python interpreter. You’ve noted that sys.executable points to your conda environment, which should be fine. However, if there’s a mismatch or an issue with how the Jupyter notebook environment is set up, consider reviewing or adjusting it. You can explicitly set the Python executable wandb should use with the _executable setting in wandb.init():
wandb.init(settings=wandb.Settings(_executable='<executable>'))
  1. Check for Environment Conflicts: Since wandb works fine from a normal Python script but not within a Jupyter notebook, there might be environment variable conflicts or differences in how the environment is set up. Do you encounter this running wandb in a fresh environment?
1 Like
3

hi @uma-wandb,

It has started working now:) I don’t think I changed anything - It looks like there were some problems with the file systems of our cluster that day so maybe it was that.

Thank you so much for your reply!