Failed to verify certificate: x509: certificate signed by unknown authority

W&B Help
1

I am running a SLURM job through my school’s cluster.
I’m able to log in to my account using wandb.login(), but, whenever I try to post to https://api.wandb.ai/graphql, there is a certification error (i.e., when using .init() or defining a logger WandbLogger() from lightning).

wandb: Using wandb-core as the SDK backend. Please refer to https://wandb.me/wandb-core for more information.
wandb: Currently logged in as: hsp287. Use `wandb login --relogin` to force relogin
wandb: Appending key for api.wandb.ai to your netrc file: /home/hsp287/.netrc
wandb: ERROR failed to upsert bucket: api: failed sending: POST https://api.wandb.ai/graphql giving up after 1 attempt(s): Post "https://api.wandb.ai/graphql": tls: failed to verify certificate: x509: certificate signed by unknown authority
2

Ok, the issue is resolved.
The issue came from using a singularity container to run the script. I think there may be some additional steps when using singularity for certification.

To solve this, I switched to using a venv and installing the needed packages directly in my project directory.

1 Like
3

Hi @hsp287, great to know you managed to fix this and thank you for sharing the fix :raised_hands: . I will mark this as resolved, please don’t hesitate to reach out in the future for any further help.

1 Like
4

Thanks. I also just wanted to follow-up if anyone wanted to use a singularity container.
The solution that worked for me was the following:

  1. Download cacert.pem from the web (https://curl.se/ca/cacert.pem) using curl or wget.
  2. In the sbatch script, export the path to the cacert.pem file:
    export SSL_CERT_FILE=/path/to/cacert.pem
1 Like
5

Still getting this in Apptainer.

6

To resolve this issue, ensure that the environment variable $SSL_CERT_FILE points to the correct certificate file.

For example, you can determine the location of the $SSL_CERT_FILE outside the container and set this variable within the container as well. If the required file is not available in the container, you can either bind mount it into the container or copy it there during the container setup process. This ensures that the container has access to the necessary certificate file for proper operation.