Artifact.add_reference('s3://...'): error in finding credentials

[jpcenteno80]

Hi, I am trying to track an artifact that lives in AWS S3. I have a ~/.aws/credentials file with two profiles and I would like to talk to AWS and store the reference with one of those profiles. I rather not use environment variables (i.e. AWS_ACCESS_KEY_ID, etc), but stick with the credentials folder and maintain the existing profiles. Are there any ways of making this work? Thanks!

Problem code:

    raw_data_uri = os.path.join('s3://', cfg.bucket_name, cfg.prefix)
    bucket = S3Bucket(cfg.bucket_name, cfg.aws_profile) # my handler to talk to S3
    num_items = len(bucket.ls(cfg.prefix))
    
    with wandb.init(project=cfg.project, job_type=cfg.job_type, name=cfg.run_stamp) as run:
        artifact = wandb.Artifact(name='raw_data', type='raw_data',
                                    metadata={
                                        'aws_profile': cfg.aws_profile,
                                        'num_items': num_items
                                        })
            
        artifact.add_reference(raw_data_uri)
        run.log_artifact(artifact)

Error:

botocore.exceptions.NoCredentialsError: Unable to locate credentials

[jpcenteno80]

Solved my own problem. Forgot I could set a temp env variable in the script itself:

os.environ['AWS_PROFILE'] = cfg.aws_profile

[_scott]

Thanks for writing an answer to your own question for any future readers!!