# OSError: Could not find a suitable TLS CA certificate **URL:** https://community.wandb.ai/t/oserror-could-not-find-a-suitable-tls-ca-certificate/3913 **Category:** W&B Help **Tags:** wandb **Created:** [February 18, 2023, 5:28pm UTC](https://community.wandb.ai/t/oserror-could-not-find-a-suitable-tls-ca-certificate/3913 "2023-02-18T17:28:30Z") **Posts on this page:** 4 **Page:** 1
### Author: ![nadzma](https://avatars.discourse-cdn.com/v4/letter/n/ac8455/32.png) [@nadzma](https://community.wandb.ai/u/nadzma) #### Post date: [February 18, 2023, 5:28pm UTC](https://community.wandb.ai/t/oserror-could-not-find-a-suitable-tls-ca-certificate/3913/1 "2023-02-18T17:28:30Z")
I am using wandb with transformers library on a conda environment created on an HPC machine. When I create the environment and install the libraries, the certificate exists under the path /conda\_environments//lib/python3.8/site-packages/certifi/cacert.pem Then after a specific time of training monitoring (around 10 hours), the certificate disappears, and I get the following error, and then the training and the monitoring are stopped. OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /scratch/hpc//conda\_environments//lib/python3.8/site-packages/certifi/cacert.pem wandb: ERROR Internal wandb error: file data was not synced There is a similar issue on the GitHub repo of wandb > > > wandb 0.10.10 > Python 3.8.3 > Debian GNU/Linux 9 (stretch) > > Hello, > … > It seems that whenever I have a model running after a certain number of hours or iterations (in my case, it's beyond 13 hours), my script gets a problem related to some TLS CA certificate. So far, I have not found an issue similar to mine in this repository. I run my script inside a tmux pane. The output is shown below. For some reason, the process did not crash or stop, I had to terminate it manually. > > > > \`\`\` > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 643, in send \[1814/1814\] > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 416, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert\_verify > wandb.errors.error.CommError: Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 101, in send\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 144, in send\_request\_status > Message: 'Failed to check stop requested status: %s' > Arguments: (CommError('Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem'),) > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: \[1759/1814\] > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 203, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/handler.py", line 48, in handle > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/handler.py", line 55, in handle\_request > Message: 'handle\_request: status' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: request' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush \[1704/1814\] > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 99, in send\_request > Message: 'send\_request: status' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 768, in check\_stop\_requested > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/old/retry.py", line 96, in \_\_call\_\_ > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 130, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in \_get\_result > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 119, in post > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 61, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 530, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 643, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 416, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert\_verify > OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 140, in send\_request\_status > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 62, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/six.py", line 702, in reraise > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 768, in check\_stop\_requested > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/old/retry.py", line 96, in \_\_call\_\_ > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 130, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in \_get\_result > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 119, in post > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 61, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 530, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 643, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 416, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert\_verify > wandb.errors.error.CommError: Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied \[1649/1814\] > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 101, in send\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 144, in send\_request\_status > Message: 'Failed to check stop requested status: %s' > Arguments: (CommError('Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem'),) > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): \[1594/1814\] > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 203, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/handler.py", line 48, in handle > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/handler.py", line 55, in handle\_request > Message: 'handle\_request: status' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: request' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 99, in send\_request > Message: 'send\_request: status' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 768, in check\_stop\_requested > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/old/retry.py", line 96, in \_\_call\_\_ > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 130, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in \_get\_result > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 119, in post > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 61, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 530, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 643, in send \[1526/1801\] > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 416, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert\_verify > OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 140, in send\_request\_status > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 62, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/six.py", line 702, in reraise > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 768, in check\_stop\_requested > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/old/retry.py", line 96, in \_\_call\_\_ > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 130, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in \_get\_result > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 119, in post > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 61, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 530, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 643, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 416, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert\_verify > wandb.errors.error.CommError: Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 101, in send\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 144, in send\_request\_status > Message: 'Failed to check stop requested status: %s' > Arguments: (CommError('Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem'),) > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send \[1458/1801\] > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 203, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/handler.py", line 48, in handle > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/handler.py", line 55, in handle\_request > Message: 'handle\_request: status' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: request' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 99, in send\_request > Message: 'send\_request: status' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 768, in check\_stop\_requested > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/old/retry.py", line 96, in \_\_call\_\_ > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 130, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in \_get\_result \[1390/1801\] > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 119, in post > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 61, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 530, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 643, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 416, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert\_verify > OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 140, in send\_request\_status > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 62, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/six.py", line 702, in reraise > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 768, in check\_stop\_requested > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/old/retry.py", line 96, in \_\_call\_\_ > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 130, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in \_get\_result > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 119, in post > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 61, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 530, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 643, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 416, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert\_verify > wandb.errors.error.CommError: Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 101, in send\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 144, in send\_request\_status > Message: 'Failed to check stop requested status: %s' > Arguments: (CommError('Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem'),) > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap \[1322/1801\] > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 203, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/handler.py", line 48, in handle > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/handler.py", line 55, in handle\_request > Message: 'handle\_request: status' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: request' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush \[1254/1801\] > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 99, in send\_request > Message: 'send\_request: status' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 768, in check\_stop\_requested > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/old/retry.py", line 96, in \_\_call\_\_ > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 130, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in \_get\_result > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 119, in post > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 61, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 530, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 643, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 416, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert\_verify > OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 140, in send\_request\_status > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 62, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/six.py", line 702, in reraise > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 768, in check\_stop\_requested > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/old/retry.py", line 96, in \_\_call\_\_ > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 130, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in \_get\_result > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 119, in post > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 61, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 530, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 643, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 416, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert\_verify > wandb.errors.error.CommError: Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 101, in send\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 144, in send\_request\_status > Message: 'Failed to check stop requested status: %s' > Arguments: (CommError('Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem'),) > \--- Logging error --- > Traceback (most recent call last): \[1186/1801\] > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 203, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/handler.py", line 48, in handle > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/handler.py", line 55, in handle\_request > Message: 'handle\_request: status' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: request' \[1118/1801\] > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 99, in send\_request > Message: 'send\_request: status' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 768, in check\_stop\_requested > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/old/retry.py", line 96, in \_\_call\_\_ > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 130, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in \_get\_result > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 119, in post > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 61, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 530, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 643, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 416, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert\_verify > OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 140, in send\_request\_status > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 62, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/six.py", line 702, in reraise > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 768, in check\_stop\_requested > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/old/retry.py", line 96, in \_\_call\_\_ > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 130, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in \_get\_result > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 119, in post > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 61, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 530, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 643, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 416, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert\_verify > wandb.errors.error.CommError: Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 101, in send\_request \[1050/1801\] > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 144, in send\_request\_status > Message: 'Failed to check stop requested status: %s' > Arguments: (CommError('Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem'),) > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 203, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/handler.py", line 48, in handle > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/handler.py", line 55, in handle\_request > Message: 'handle\_request: status' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner \[982/1801\] > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: request' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 99, in send\_request > Message: 'send\_request: status' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 768, in check\_stop\_requested > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/old/retry.py", line 96, in \_\_call\_\_ > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 130, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in \_get\_result > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 119, in post > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 61, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 530, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 643, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 416, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert\_verify > OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 140, in send\_request\_status > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 62, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/six.py", line 702, in reraise > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 768, in check\_stop\_requested > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/old/retry.py", line 96, in \_\_call\_\_ > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 130, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in \_get\_result > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 119, in post > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 61, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 530, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 643, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 416, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert\_verify > wandb.errors.error.CommError: Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner \[914/1801\] > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 101, in send\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 144, in send\_request\_status > Message: 'Failed to check stop requested status: %s' > Arguments: (CommError('Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem'),) > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush \[846/1801\] > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 203, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/handler.py", line 48, in handle > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/handler.py", line 55, in handle\_request > Message: 'handle\_request: status' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: request' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 99, in send\_request > Message: 'send\_request: status' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 768, in check\_stop\_requested > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/old/retry.py", line 96, in \_\_call\_\_ > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 130, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in \_get\_result > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 119, in post > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 61, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 530, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 643, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 416, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert\_verify > OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 140, in send\_request\_status > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 62, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/six.py", line 702, in reraise > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 768, in check\_stop\_requested > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/old/retry.py", line 96, in \_\_call\_\_ > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 130, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute \[778/1801\] > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in \_get\_result > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 119, in post > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 61, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 530, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 643, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 416, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert\_verify > wandb.errors.error.CommError: Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 101, in send\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 144, in send\_request\_status > Message: 'Failed to check stop requested status: %s' > Arguments: (CommError('Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem'),) > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' \[710/1801\] > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 203, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/handler.py", line 48, in handle > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/handler.py", line 55, in handle\_request > Message: 'handle\_request: status' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: request' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 99, in send\_request > Message: 'send\_request: status' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 768, in check\_stop\_requested > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/old/retry.py", line 96, in \_\_call\_\_ > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 130, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in \_get\_result > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 119, in post > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 61, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 530, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 643, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 416, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert\_verify > OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 140, in send\_request\_status > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 62, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/six.py", line 702, in reraise \[642/1801\] > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 768, in check\_stop\_requested > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/old/retry.py", line 96, in \_\_call\_\_ > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 130, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in \_get\_result > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 119, in post > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 61, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 530, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 643, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 416, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert\_verify > wandb.errors.error.CommError: Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 101, in send\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 144, in send\_request\_status > Message: 'Failed to check stop requested status: %s' > Arguments: (CommError('Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem'),) > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner \[574/1801\] > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 203, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/handler.py", line 48, in handle > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/handler.py", line 55, in handle\_request > Message: 'handle\_request: status' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: request' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 99, in send\_request > Message: 'send\_request: status' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 768, in check\_stop\_requested \[506/1801\] > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/old/retry.py", line 96, in \_\_call\_\_ > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 130, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in \_get\_result > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 119, in post > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 61, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 530, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 643, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 416, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert\_verify > OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 140, in send\_request\_status > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 62, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/six.py", line 702, in reraise > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 768, in check\_stop\_requested > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/old/retry.py", line 96, in \_\_call\_\_ > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 130, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in \_get\_result > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 119, in post > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 61, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 530, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 643, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 416, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert\_verify > wandb.errors.error.CommError: Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 101, in send\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 144, in send\_request\_status > Message: 'Failed to check stop requested status: %s' > Arguments: (CommError('Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem'),) > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit \[438/1801\] > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: stats' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 203, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/handler.py", line 48, in handle > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/handler.py", line 55, in handle\_request > Message: 'handle\_request: status' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: request' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 99, in send\_request > Message: 'send\_request: status' \[370/1801\] > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 768, in check\_stop\_requested > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/old/retry.py", line 96, in \_\_call\_\_ > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 130, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in \_get\_result > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 119, in post > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 61, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 530, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 643, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 416, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert\_verify > OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 140, in send\_request\_status > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 62, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/six.py", line 702, in reraise > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/apis/normalize.py", line 24, in wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 768, in check\_stop\_requested > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/old/retry.py", line 96, in \_\_call\_\_ > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_api.py", line 130, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 52, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/client.py", line 60, in \_get\_result > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/vendor/gql-0.2.0/gql/transport/requests.py", line 38, in execute > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 119, in post > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/api.py", line 61, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 530, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/sessions.py", line 643, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 416, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/requests/adapters.py", line 227, in cert\_verify > wandb.errors.error.CommError: Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 101, in send\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 144, in send\_request\_status > Message: 'Failed to check stop requested status: %s' > Arguments: (CommError('Could not find a suitable TLS CA certificate bundle, invalid path: /u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/certifi/cacert.pem'),) > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send \[302/1801\] > Message: 'send: stats' > Arguments: () > Validation loss: 0.1446361123191008 > > Exception in thread stdout: > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: history' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 90, in send > Message: 'send: summary' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 61, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 260, in \_finish > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/writer.py", line 38, in finish > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/datastore.py", line 257, in close > Message: 'close: %s' > Arguments: ('/u/salaunol/Documents/\_2020\_automne/1\_PREDICT\_ARTICLES/wandb/run-20201111\_132925-2qrgmr2k/run-2qrgmr2k.wandb',) > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 61, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 206, in \_finish > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/handler.py", line 271, in finish > Message: 'shutting down handler' > Arguments: () > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: \[234/1801\] > File "\", line 1, in \ > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn\_main > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/multiprocessing/spawn.py", line 129, in \_main > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/multiprocessing/process.py", line 315, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/multiprocessing/process.py", line 108, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 129, in wandb\_internal > Message: 'Thread SenderThread:' > Arguments: () > Thread SenderThread: > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 33, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal\_util.py", line 60, in \_run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 233, in \_process > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 92, in send > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/sender.py", line 509, in send\_summary > OSError: \[Errno 127\] Key has expired: '/u/salaunol/Documents/\_2020\_automne/1\_PREDICT\_ARTICLES/wandb/run-20201111\_132925-2qrgmr2k/files/wandb-summary.json' > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 870, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/worker.py", line 137, in \_target > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 311, in send\_event\_wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 229, in \_send\_event > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 174, in \_send\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/request.py", line 79, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/request.py", line 171, in request\_encode\_body > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/poolmanager.py", line 336, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 659, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 279, in \_get\_conn > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 937, in \_new\_conn > Message: 'Starting new HTTPS connection (%d): %s:%s' > Arguments: (1, 'o151352.ingest.sentry.io', 443) > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/util/ssl\_.py", line 343, in ssl\_wrap\_socket > OSError: \[Errno 127\] Key has expired > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 381, in \_make\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 976, in \_validate\_conn > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connection.py", line 361, in connect > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/util/ssl\_.py", line 345, in ssl\_wrap\_socket > urllib3.exceptions.SSLError: \[Errno 127\] Key has expired > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 870, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/worker.py", line 137, in \_target > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 311, in send\_event\_wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 229, in \_send\_event > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 174, in \_send\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/request.py", line 79, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/request.py", line 171, in request\_encode\_body \[166/1801\] > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/poolmanager.py", line 336, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 724, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/util/retry.py", line 441, in increment > Message: "Incremented Retry for (url='%s'): %r" > Arguments: ('/api/5288891/store/', Retry(total=2, connect=None, read=None, redirect=None, status=None)) > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 870, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/worker.py", line 137, in \_target > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 311, in send\_event\_wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 229, in \_send\_event > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 174, in \_send\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/request.py", line 79, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/request.py", line 171, in request\_encode\_body > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/poolmanager.py", line 336, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 749, in urlopen > Message: "Retrying (%r) after connection broken by '%r': %s" > Arguments: (Retry(total=2, connect=None, read=None, redirect=None, status=None), SSLError(OSError(127, 'Key has expired')), '/api/5288891/store/') > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 870, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/worker.py", line 137, in \_target > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 311, in send\_event\_wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 229, in \_send\_event > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 174, in \_send\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/request.py", line 79, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/request.py", line 171, in request\_encode\_body > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/poolmanager.py", line 336, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 752, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 659, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 279, in \_get\_conn > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 937, in \_new\_conn > Message: 'Starting new HTTPS connection (%d): %s:%s' > Arguments: (2, 'o151352.ingest.sentry.io', 443) > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/util/ssl\_.py", line 343, in ssl\_wrap\_socket > OSError: \[Errno 127\] Key has expired > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 381, in \_make\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 976, in \_validate\_conn > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connection.py", line 361, in connect > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/util/ssl\_.py", line 345, in ssl\_wrap\_socket > urllib3.exceptions.SSLError: \[Errno 127\] Key has expired > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: \[98/1801\] > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 870, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/worker.py", line 137, in \_target > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 311, in send\_event\_wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 229, in \_send\_event > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 174, in \_send\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/request.py", line 79, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/request.py", line 171, in request\_encode\_body > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/poolmanager.py", line 336, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 752, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 724, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/util/retry.py", line 441, in increment > Message: "Incremented Retry for (url='%s'): %r" > Arguments: ('/api/5288891/store/', Retry(total=1, connect=None, read=None, redirect=None, status=None)) > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 870, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/worker.py", line 137, in \_target > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 311, in send\_event\_wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 229, in \_send\_event > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 174, in \_send\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/request.py", line 79, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/request.py", line 171, in request\_encode\_body > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/poolmanager.py", line 336, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 752, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 749, in urlopen > Message: "Retrying (%r) after connection broken by '%r': %s" > Arguments: (Retry(total=1, connect=None, read=None, redirect=None, status=None), SSLError(OSError(127, 'Key has expired')), '/api/5288891/store/') > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 870, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/worker.py", line 137, in \_target > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 311, in send\_event\_wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 229, in \_send\_event > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 174, in \_send\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/request.py", line 79, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/request.py", line 171, in request\_encode\_body > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/poolmanager.py", line 336, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 752, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 752, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 659, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 279, in \_get\_conn > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 937, in \_new\_conn > Message: 'Starting new HTTPS connection (%d): %s:%s' > Arguments: (3, 'o151352.ingest.sentry.io', 443) > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/util/ssl\_.py", line 343, in ssl\_wrap\_socket > OSError: \[Errno 127\] Key has expired > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 381, in \_make\_request \[30/1801\] > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 976, in \_validate\_conn > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connection.py", line 361, in connect > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/util/ssl\_.py", line 345, in ssl\_wrap\_socket > urllib3.exceptions.SSLError: \[Errno 127\] Key has expired > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 870, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/worker.py", line 137, in \_target > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 311, in send\_event\_wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 229, in \_send\_event > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 174, in \_send\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/request.py", line 79, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/request.py", line 171, in request\_encode\_body > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/poolmanager.py", line 336, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 752, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 752, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 724, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/util/retry.py", line 441, in increment > Message: "Incremented Retry for (url='%s'): %r" > Arguments: ('/api/5288891/store/', Retry(total=0, connect=None, read=None, redirect=None, status=None)) > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 870, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/worker.py", line 137, in \_target > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 311, in send\_event\_wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 229, in \_send\_event > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 174, in \_send\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/request.py", line 79, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/request.py", line 171, in request\_encode\_body > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/poolmanager.py", line 336, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 752, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 752, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 749, in urlopen > Message: "Retrying (%r) after connection broken by '%r': %s" > Arguments: (Retry(total=0, connect=None, read=None, redirect=None, status=None), SSLError(OSError(127, 'Key has expired')), '/api/5288891/store/') > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 890, in \_bootstrap > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 932, in \_bootstrap\_inner > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/threading.py", line 870, in run > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/worker.py", line 137, in \_target > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 311, in send\_event\_wrapper > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 229, in \_send\_event > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/sentry\_sdk/transport.py", line 174, in \_send\_request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/request.py", line 79, in request > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/request.py", line 171, in request\_encode\_body > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/poolmanager.py", line 336, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 752, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 752, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 752, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 659, in urlopen > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 279, in \_get\_conn > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/urllib3/connectionpool.py", line 937, in \_new\_conn > Message: 'Starting new HTTPS connection (%d): %s:%s' > Arguments: (4, 'o151352.ingest.sentry.io', 443) > \--- Logging error --- > Traceback (most recent call last): > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1085, in emit > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/logging/\_\_init\_\_.py", line 1065, in flush > PermissionError: \[Errno 13\] Permission denied > Call stack: > File "/u/salaunol/anaconda3/envs/cuda101/lib/python3.8/site-packages/wandb/sdk/internal/internal.py", line 138, in handle\_exit > Message: 'Internal process exited' > Arguments: () > \`\`\` I reproduced this issue several times with different models and different pipelines to check that it was not something related to my code. The common pattern is that this happened only for long training (more than 10 hours) Thank you ---
### Author: ![raphael-sanandres](https://avatars.discourse-cdn.com/v4/letter/r/7cd45c/32.png) [@raphael-sanandres](https://community.wandb.ai/u/raphael-sanandres) #### Post date: [February 23, 2023, 8:15pm UTC](https://community.wandb.ai/t/oserror-could-not-find-a-suitable-tls-ca-certificate/3913/2 "2023-02-23T20:15:16Z")
Hello Nadhem! Just some questions to get a better understanding of the issue: - Are you on Public Cloud or a Local Instance? - If you are on a Local Instance, could you send the Debug Bundle? An admin of the instance can get it from the `/system-admin` page → top right corner W&B icon → `Debug Bundle`. - If you are on the Public Cloud, could you send your `debug.log` and `debug-internal.log` for the run? To get the `debug.log` and `debug-internal.log` files, go to the wandb folder in your computer’s working directory. The folder has subfolders named run-DATETIME-ID, which correspond to specific runs. Could you retrieve the debug logs for a run that stopped? ---
### Author: ![raphael-sanandres](https://avatars.discourse-cdn.com/v4/letter/r/7cd45c/32.png) [@raphael-sanandres](https://community.wandb.ai/u/raphael-sanandres) #### Post date: [February 28, 2023, 11:05pm UTC](https://community.wandb.ai/t/oserror-could-not-find-a-suitable-tls-ca-certificate/3913/3 "2023-02-28T23:05:22Z")
Hi there, I wanted to follow up on this request. Please let us know if we can be of further assistance or if your issue has been resolved. ---
### Author: ![system](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/1X/366b649231631dbab896843020da0056074ac79d.png) [@system](https://community.wandb.ai/u/system) #### Post date: [April 19, 2023, 5:28pm UTC](https://community.wandb.ai/t/oserror-could-not-find-a-suitable-tls-ca-certificate/3913/4 "2023-04-19T17:28:50Z")
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.