# wandb.errors.Error: An unexpected error occurred **URL:** \ **Category:** W&B Help\ **Tags:** wandb\ **Created:** [May 25, 2024, 11:20pm UTC](https://community.wandb.ai/t/wandb-errors-error-an-unexpected-error-occurred/6780 "2024-05-25T23:20:08Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![seyuboglu](https://avatars.discourse-cdn.com/v4/letter/s/e47774/32.png) [@seyuboglu](https://community.wandb.ai/u/seyuboglu)\ **Post date:** [May 25, 2024, 11:20pm UTC](https://community.wandb.ai/t/wandb-errors-error-an-unexpected-error-occurred/6780/1 "2024-05-25T23:20:08Z")
When I launch wandb on some (but not all) of the nodes in my cluster I get the following error. Moving to another machine resolves the issue, but I can’t launch wandb on that node even though they are using the exact same conda environment and wandb installation. ```auto File ".../lib/python3.11/site-packages/pytorch_lightning/loggers/wandb.py", line 360, in __init__ _ = self.experiment ^^^^^^^^^^^^^^^ File ".../lib/python3.11/site-packages/pytorch_lightning/loggers/logger.py", line 53, in experiment return get_experiment() or DummyExperiment() ^^^^^^^^^^^^^^^^ File ".../lib/python3.11/site-packages/lightning_utilities/core/rank_zero.py", line 43, in wrapped_fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File ".../lib/python3.11/site-packages/pytorch_lightning/loggers/logger.py", line 51, in get_experiment return fn(self) ^^^^^^^^ File "/lib/python3.11/site-packages/pytorch_lightning/loggers/wandb.py", line 406, in experiment self._experiment = wandb.init(**self._wandb_init) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/lib/python3.11/site-packages/wandb/sdk/wandb_init.py", line 1219, in init raise Error("An unexpected error occurred") from error_seen wandb.errors.Error: An unexpected error occurred ``` When I reproduce the issue outside of pytorch lightning, I get the following error. However, I’m unable to get permissions for that folder. ```auto PermissionError: /tmp/wandb/run-20240525_232234-8un2rikz/logs is not writable ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/1X/366b649231631dbab896843020da0056074ac79d.png) [@system](https://community.wandb.ai/u/system)\ **Post date:** [May 25, 2024, 11:23pm UTC](https://community.wandb.ai/t/wandb-errors-error-an-unexpected-error-occurred/6780/2 "2024-05-25T23:23:19Z")
Hello, This a reply from our support bot designed to assist you with your Weights & Biases related queries. To reach a human please reply to this message. The error you’re encountering, `wandb.errors.Error: An unexpected error occurred`, during the initialization of `wandb` with PyTorch Lightning on some nodes of your cluster, suggests there might be an issue with the environment or configuration on those specific nodes. Here are some steps and best practices to troubleshoot and potentially resolve this issue: ### 1. Check `wandb` Version Compatibility Ensure that the version of `wandb` you’re using is compatible with your version of PyTorch Lightning. It’s a good practice to keep both libraries up to date. If necessary, update `wandb` and PyTorch Lightning to their latest versions. ### 2. Verify Network Configuration Since the issue occurs on specific nodes, there might be a network configuration issue that prevents `wandb` from communicating with its servers. Verify that the nodes in question have proper internet access and that any firewall or proxy settings allow `wandb` to make outbound connections. ### 3. Environment Variables Ensure that the `WANDB_API_KEY` is set correctly on all nodes. The API key is required for `wandb` to authenticate with its servers. You can set it in your environment variables or pass it directly in your script. ### 4. Debugging Mode Enable `wandb`’s debug mode to get more detailed logs that might pinpoint the issue. You can do this by setting the environment variable `WANDB_DEBUG` to `true`. This might provide more insights into what’s going wrong during initialization. ```python import os os.environ["WANDB_DEBUG"] = "true" ``` ### 5. Offline Mode If network issues are suspected, try running `wandb` in offline mode to see if the initialization succeeds. This can help determine if the problem is network-related. You can enable offline mode by setting the environment variable `WANDB_MODE` to `offline`. ```python import os os.environ["WANDB_MODE"] = "offline" ``` ### 6. Clean Installation On the nodes where you’re experiencing issues, try creating a fresh virtual environment and reinstall `wandb` and PyTorch Lightning. This can help rule out any issues related to corrupted installations or conflicts with other packages. ### 7. Check for Errors in PyTorch Lightning Integration Ensure that the integration code between `wandb` and PyTorch Lightning is correctly set up. Misconfigurations or outdated integration code can lead to initialization errors. ### 8. Consult Logs and Documentation Review the detailed logs generated by enabling `WANDB_DEBUG` and consult the `wandb` and PyTorch Lightning documentation for any known issues or additional troubleshooting steps. If after following these steps the issue persists, consider reaching out to the `wandb` support team with detailed logs and information about your setup for further assistance. To reach a human please reply to this message. -WandBot 🤖 ---
**Author:** ![paulo-sabile](https://avatars.discourse-cdn.com/v4/letter/p/f14d63/32.png) [@paulo-sabile](https://community.wandb.ai/u/paulo-sabile)\ **Post date:** [May 31, 2024, 6:37am UTC](https://community.wandb.ai/t/wandb-errors-error-an-unexpected-error-occurred/6780/3 "2024-05-31T06:37:19Z")
Hi @seyuboglu Good day and thank you for reaching out to us. Happy to help you here. The error appears to be a permission issue and since you mentioned that it works just fine once you move to another machine, then this somehow suggests that it is. Are you using a local instance of wandb? `PermissionError: /tmp/wandb/run-20240525_232234-8un2rikz/logs is not writable` Are you setting this path yourself? If so, can you let us know how you are doing it? On the machine where you are getting errors, please ensure that the version of `wandb` you’re using is the same with the version from the other machine that has no errors. Let us see if this is only caused by a version mismatch. I’m looking forward for your next update. Best Regards, Paulo ---
**Author:** ![paulo-sabile](https://avatars.discourse-cdn.com/v4/letter/p/f14d63/32.png) [@paulo-sabile](https://community.wandb.ai/u/paulo-sabile)\ **Post date:** [June 4, 2024, 9:26am UTC](https://community.wandb.ai/t/wandb-errors-error-an-unexpected-error-occurred/6780/4 "2024-06-04T09:26:03Z")
Hi @seyuboglu , We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved. Best, Weights & Biases ---
**Author:** ![paulo-sabile](https://avatars.discourse-cdn.com/v4/letter/p/f14d63/32.png) [@paulo-sabile](https://community.wandb.ai/u/paulo-sabile)\ **Post date:** [June 7, 2024, 3:22am UTC](https://community.wandb.ai/t/wandb-errors-error-an-unexpected-error-occurred/6780/5 "2024-06-07T03:22:18Z")
Hi @seyuboglu, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!