Wandb.finish() crashes

I’m getting an error when I try to call wandb.finish()

This is also the case when I switch to an older commit.
The bug is also triggered when I simply run the following:

import wandb
wandb.init(project="test", entity="tim-kuipers", job_type="train")
wandb.log({"acc": 0.9, "loss": 1.0})
wandb.finish()

My WandB seems to be in a broken state and it cannot escape.

I tried killing the wandb service, so that it would restart, but to no avail.

I tried deleting the last couple of runs.

How do I clean my working directory and get wandb working again?

Here’s the error:

TypeError                                 Traceback (most recent call last)
Cell In[5], line 1
----> 1 wandb.finish()

File ~/mambaforge-pypy3/envs/fastai_env/lib/python3.10/site-packages/wandb/sdk/wandb_run.py:3711, in finish(exit_code, quiet)
   3701 """Mark a run as finished, and finish uploading all data.
   3702 
   3703 This is used when creating multiple runs in the same process.
   (...)
   3708     quiet: Set to true to minimize log output
   3709 """
   3710 if wandb.run:
-> 3711     wandb.run.finish(exit_code=exit_code, quiet=quiet)

File ~/mambaforge-pypy3/envs/fastai_env/lib/python3.10/site-packages/wandb/sdk/wandb_run.py:390, in _run_decorator._noop..wrapper(self, *args, **kwargs)
    387         wandb.termwarn(message, repeat=False)
    388         return cls.Dummy()
--> 390 return func(self, *args, **kwargs)

File ~/mambaforge-pypy3/envs/fastai_env/lib/python3.10/site-packages/wandb/sdk/wandb_run.py:331, in _run_decorator._attach..wrapper(self, *args, **kwargs)
    329         raise e
    330     cls._is_attaching = ""
--> 331 return func(self, *args, **kwargs)

File ~/mambaforge-pypy3/envs/fastai_env/lib/python3.10/site-packages/wandb/sdk/wandb_run.py:1879, in Run.finish(self, exit_code, quiet)
   1865 @_run_decorator._noop
   1866 @_run_decorator._attach
   1867 def finish(
   1868     self, exit_code: Optional[int] = None, quiet: Optional[bool] = None
   1869 ) -> None:
   1870     """Mark a run as finished, and finish uploading all data.
   1871 
   1872     This is used when creating multiple runs in the same process. We automatically
   (...)
   1877         quiet: Set to true to minimize log output
   1878     """
-> 1879     return self._finish(exit_code, quiet)

File ~/mambaforge-pypy3/envs/fastai_env/lib/python3.10/site-packages/wandb/sdk/wandb_run.py:1892, in Run._finish(self, exit_code, quiet)
   1890 for hook in self._teardown_hooks:
   1891     if hook.stage == TeardownStage.EARLY:
-> 1892         hook.call()
   1894 self._atexit_cleanup(exit_code=exit_code)
   1895 if self._wl and len(self._wl._global_run_stack) > 0:

File ~/mambaforge-pypy3/envs/fastai_env/lib/python3.10/site-packages/wandb/sdk/wandb_init.py:430, in _WandbInit._jupyter_teardown(self)
    428 assert self.notebook
    429 ipython = self.notebook.shell
--> 430 self.notebook.save_history()
    431 if self.notebook.save_ipynb():
    432     assert self.run is not None

File ~/mambaforge-pypy3/envs/fastai_env/lib/python3.10/site-packages/wandb/jupyter.py:435, in Notebook.save_history(self)
    433 """This saves all cell executions in the current session as a new notebook."""
    434 try:
--> 435     from nbformat import v4, validator, write
    436 except ImportError:
    437     logger.error("Run pip install nbformat to save notebook history")

File ~/mambaforge-pypy3/envs/fastai_env/lib/python3.10/site-packages/nbformat/__init__.py:12
      9 from traitlets.log import get_logger
     11 from . import v1, v2, v3, v4
---> 12 from ._version import __version__, version_info
     13 from .sentinel import Sentinel
     15 __all__ = [
     16     "versions",
     17     "validate",
   (...)
     32     "Sentinel",
     33 ]

File ~/mambaforge-pypy3/envs/fastai_env/lib/python3.10/site-packages/nbformat/_version.py:28
     12 # matches tbump regex in pyproject.toml
     13 _version_regex = re.compile(
     14     r"""
     15   (?P\d+)
   (...)
     25     re.VERBOSE,
     26 )
---> 28 _version_fields = _version_regex.match(__version__).groupdict()  # type:ignore
     29 version_info = tuple(
     30     field
     31     for field in (
   (...)
     38     if field is not None
     39 )

TypeError: expected string or bytes-like object

Not sure if this helps:

wandb verify
Default host selected: https://api.wandb.ai
Find detailed logs for this test at: /tmp/tmpoolua1ny/wandb
❌
Cannot run wandb verify against api.wandb.ai

There are no logs in that location. The dir is empty.

Hey @tim-kuipers, are you using Launch to run this? To troubleshoot this, I recommend trying to run this in a separate environment with only wandb and jupyter installed. Please let me know how this goes.

Additionally, according to the traceback, I noticed the following chunk:

    434 try:
--> 435     from nbformat import v4, validator, write
    436 except ImportError:
    437     logger.error("Run pip install nbformat to save notebook history")

I recommend also trying to install nbformat after you try the above troubleshooting suggestion.

You’re reading the error message wrong. The error is happening inside nbformat, so this means I already have nbformat installed.

Somehow the error doesn’t reproduce today. I am working remotely through ssh, but will try locally tomorrow to see if the issue really has been resolved.

Hey @tim-kuipers, were you able to reproduce the error? Additionally, I would love to know the differences in behavior between running this remotely vs locally. Any error messages would be incredibly helpful here, and I would love to take a look at your debug.log​and debug-internal.log files. They should be located in the wandb folder in the same directory as where the script was run. The wandb folder has folders formatted as run-DATETIME-ID associated with a single run.

It was weird. The error did reproduce locally once or twice (I think), but it resolved itself somehow. I’m not sure I did anything to resolve it. Maybe I updated some package or something. Wish I could help others facing this problem, but for me the problem was somehow resolved and I don’t know how.

@tim-kuipers Glad to hear it was resolved, but feel free to open this thread again or start a new one if the problem arises again. Would be happy to troubleshoot then! I’ll close out this ticket for now; have a great week!

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.