# Every wandb command yields "An error occured in MPI\_Init\_thread" **URL:** \ **Category:** W&B Help\ **Tags:** wandb\ **Created:** [November 15, 2022, 11:40pm UTC](https://community.wandb.ai/t/every-wandb-command-yields-an-error-occured-in-mpi-init-thread/3422 "2022-11-15T23:40:42Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![andravin](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@andravin](https://community.wandb.ai/u/andravin)\ **Post date:** [November 15, 2022, 11:40pm UTC](https://community.wandb.ai/t/every-wandb-command-yields-an-error-occured-in-mpi-init-thread/3422/1 "2022-11-15T23:40:42Z")
I have a user on a multi-user server whose wandb installation appears to be broken. I have forced him to reinstall with ```auto python3 -m pip install --upgrade pip python3 -m pip install --upgrade --force-reinstall wandb ``` after which he has wandb-0.13.5. But this error persists; even the simplest `wandb` command yields an `MPI_Init_thread` error: ```auto $ wandb --version *** An error occurred in MPI_Init_thread *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [xhostnamex:1359609] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed! ``` On the same machine, my `wandb` installation works fine: ```auto $ wandb --version wandb, version 0.13.5 ``` Apologies if this is a duplicate, but I could not find this exact problem documented anywhere. ---
**Author:** ![ramit\_goolry](https://avatars.discourse-cdn.com/v4/letter/r/85e7bf/32.png) [@ramit\_goolry](https://community.wandb.ai/u/ramit_goolry)\ **Post date:** [November 16, 2022, 6:01am UTC](https://community.wandb.ai/t/every-wandb-command-yields-an-error-occured-in-mpi-init-thread/3422/2 "2022-11-16T06:01:02Z")
Hey @andravin, Is this the full stack trace for the error? If so, please run `wandb.init()` in a python file and share the full error trace from there along with the `debug.log` and `debug-internal.log` files from the `wandb` folder in the directory where you rant this process. Thanks, Ramit ---
**Author:** ![moshin](https://avatars.discourse-cdn.com/v4/letter/m/45deac/32.png) [@moshin](https://community.wandb.ai/u/moshin)\ **Post date:** [November 18, 2022, 1:37am UTC](https://community.wandb.ai/t/every-wandb-command-yields-an-error-occured-in-mpi-init-thread/3422/3 "2022-11-18T01:37:52Z")
Hi @ramit_goolry , I’m the user who @andravin mentioned. Here is what I did ```auto moshin@pc$ python -V Python 3.8.10 moshin@pc$ ls testwandb.py moshin@pc$ cat testwandb.py import wandb wandb.init() moshin@pc$ python testwandb.py *** An error occurred in MPI_Init_thread *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [pc:1623678] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed! moshin@pc$ ls testwandb.py ``` It seems the process is broken before `wandb` folder is created. ---
**Author:** ![andravin](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@andravin](https://community.wandb.ai/u/andravin)\ **Post date:** [December 5, 2022, 3:14am UTC](https://community.wandb.ai/t/every-wandb-command-yields-an-error-occured-in-mpi-init-thread/3422/4 "2022-12-05T03:14:12Z")
We determined the error occurs on every node in the cluster, but only when the user session is created with [SLURM](https://slurm.schedmd.com/overview.html). A typical command-line to create the SLURM interactive session is `srun --pty bash`. The error is always reproducible simply by `import wandb`: ```auto $ python3 -c 'import wandb' *** An error occurred in MPI_Init_thread *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [node-name:1431860] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed! ``` If the user connects to the node with `ssh`, then `import wandb` works as expected. ---
**Author:** ![andravin](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@andravin](https://community.wandb.ai/u/andravin)\ **Post date:** [December 5, 2022, 5:31am UTC](https://community.wandb.ai/t/every-wandb-command-yields-an-error-occured-in-mpi-init-thread/3422/5 "2022-12-05T05:31:15Z")
Also, a different user with the same exact python packages installed via `pip` does not have this issue. ---
**Author:** ![ramit\_goolry](https://avatars.discourse-cdn.com/v4/letter/r/85e7bf/32.png) [@ramit\_goolry](https://community.wandb.ai/u/ramit_goolry)\ **Post date:** [December 6, 2022, 4:49am UTC](https://community.wandb.ai/t/every-wandb-command-yields-an-error-occured-in-mpi-init-thread/3422/6 "2022-12-06T04:49:56Z")
Hi @andravin! Any chance you can try `mpirun -np [NUM]` instead of `srun` here? This looks to be an issue with `srun` your slurm cluster specifically and not wandb, I would check your installation of `srun`/`mpirun` to make sure everything is running as expected over there. Have you tried importing any other libraries through which you also see the same error? Thanks, Ramit ---
**Author:** ![ramit\_goolry](https://avatars.discourse-cdn.com/v4/letter/r/85e7bf/32.png) [@ramit\_goolry](https://community.wandb.ai/u/ramit_goolry)\ **Post date:** [December 13, 2022, 1:59pm UTC](https://community.wandb.ai/t/every-wandb-command-yields-an-error-occured-in-mpi-init-thread/3422/7 "2022-12-13T13:59:30Z")
Hi! We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved. ---
**Author:** ![artsiom](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/artsiom/32/1001_2.png) [@artsiom](https://community.wandb.ai/u/artsiom)\ **Post date:** [December 14, 2022, 4:05pm UTC](https://community.wandb.ai/t/every-wandb-command-yields-an-error-occured-in-mpi-init-thread/3422/8 "2022-12-14T16:05:05Z")
Hi Andrew, We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved. Best, Weights & Biases ---
**Author:** ![andravin](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@andravin](https://community.wandb.ai/u/andravin)\ **Post date:** [December 16, 2022, 12:02am UTC](https://community.wandb.ai/t/every-wandb-command-yields-an-error-occured-in-mpi-init-thread/3422/9 "2022-12-16T00:02:41Z")
@ramit_goolry Thanks for the suggestion, our IT department will look into our SLURM and MPI configuration. It would appear that any issue is specific to the one particular user who sees the error, because WandB works fine for everyone else. So it would have to be a user-id inconsistency or permissions problem I would think. ---
**Author:** ![artsiom](https://sea2.discourse-cdn.com/flex020/user_avatar/community.wandb.ai/artsiom/32/1001_2.png) [@artsiom](https://community.wandb.ai/u/artsiom)\ **Post date:** [December 16, 2022, 5:04pm UTC](https://community.wandb.ai/t/every-wandb-command-yields-an-error-occured-in-mpi-init-thread/3422/10 "2022-12-16T17:04:49Z")
Sounds good! Let us know if you guys have any more concerns related to wandb! ---
**Author:** ![system](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/1X/366b649231631dbab896843020da0056074ac79d.png) [@system](https://community.wandb.ai/u/system)\ **Post date:** [February 14, 2023, 12:03am UTC](https://community.wandb.ai/t/every-wandb-command-yields-an-error-occured-in-mpi-init-thread/3422/11 "2023-02-14T00:03:19Z")
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.