# Integrating Wandb and AWS Lambda - Multiprocessing

**URL:** <https://community.wandb.ai/t/integrating-wandb-and-aws-lambda-multiprocessing/4160>\
**Category:** W&B Help\
**Tags:** artifacts\
**Created:** [March 31, 2023, 10:20am UTC](https://community.wandb.ai/t/integrating-wandb-and-aws-lambda-multiprocessing/4160 "2023-03-31T10:20:40Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![chiara\_mc](https://avatars.discourse-cdn.com/v4/letter/c/94ad74/32.png) [@chiara\_mc](https://community.wandb.ai/u/chiara_mc)\
**Post date:** [March 31, 2023, 10:20am UTC](https://community.wandb.ai/t/integrating-wandb-and-aws-lambda-multiprocessing/4160/1 "2023-03-31T10:20:40Z")

</div>

Hello!

When I try to download an artifact in an AWS Lambda, I get the following error:

```auto
[ERROR] OSError: [Errno 38] Function not implemented
Traceback (most recent call last):
  File "/var/task/on_lambda.py", line 149, in lambda_entrypoint
    generate_predictions(
  File "/var/task/on_lambda.py", line 75, in generate_predictions
    models_data = get_models_data()
  File "/var/task/on_lambda.py", line 156, in <lambda>
    get_models_data=lambda: get_customer_models_data(
  File "/var/task/models.py", line 38, in get_customer_models_data
    artifact_dir = artifact.download(root=CUSTOMER_DATA_DIR)
  File "/var/task/wandb/apis/public.py", line 3763, in download
    pool = multiprocessing.dummy.Pool(32)
  File "/var/lang/lib/python3.9/multiprocessing/dummy/ __init__.py", line 124, in Pool
    return ThreadPool(processes, initializer, initargs)
  File "/var/lang/lib/python3.9/multiprocessing/pool.py", line 927, in __init__
    Pool. __init__ (self, processes, initializer, initargs)
  File "/var/lang/lib/python3.9/multiprocessing/pool.py", line 196, in __init__
    self._change_notifier = self._ctx.SimpleQueue()
  File "/var/lang/lib/python3.9/multiprocessing/context.py", line 113, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/var/lang/lib/python3.9/multiprocessing/queues.py", line 341, in __init__
    self._rlock = ctx.Lock()
  File "/var/lang/lib/python3.9/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
  File "/var/lang/lib/python3.9/multiprocessing/synchronize.py", line 162, in __init__
    SemLock. __init__ (self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/var/lang/lib/python3.9/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(

```

Is there a way around this? It seems to be an issue with multiprocessing. I tried setting `WANDB_START_METHOD = thread` which I saw mentioned in a few places, but the error doesn’t change.

This is how I’m downloading the artifact:

```auto
    api = wandb.Api()
    artifact = api.artifact(artifact_name)
    artifact_dir = artifact.download(root=data_dir)

```

I saw two other threads on this topic but they didn’t have a solution

Thanks!

---

<div class="post-metadata">

**Author:** ![ramit\_goolry](https://avatars.discourse-cdn.com/v4/letter/r/85e7bf/32.png) [@ramit\_goolry](https://community.wandb.ai/u/ramit_goolry)\
**Post date:** [April 5, 2023, 5:01pm UTC](https://community.wandb.ai/t/integrating-wandb-and-aws-lambda-multiprocessing/4160/2 "2023-04-05T17:01:25Z")

</div>

Hi @chiara_mc,

AWS Lambda does not support the use of Semaphores at the moment : [python multiprocessing - \_multiprocessing.SemLock is not implemented when running on AWS Lambda - Stack Overflow](https://stackoverflow.com/questions/34005930/multiprocessing-semlock-is-not-implemented-when-running-on-aws-lambda)

Is there a specific use case you have in mind? We can see if there is any way to work around this issue since the direct usage of artifacts does not seem possible.

Thanks,  
Ramit

---

<div class="post-metadata">

**Author:** ![ramit\_goolry](https://avatars.discourse-cdn.com/v4/letter/r/85e7bf/32.png) [@ramit\_goolry](https://community.wandb.ai/u/ramit_goolry)\
**Post date:** [April 8, 2023, 1:39am UTC](https://community.wandb.ai/t/integrating-wandb-and-aws-lambda-multiprocessing/4160/3 "2023-04-08T01:39:35Z")

</div>

Hi @chiara_mc, I wanted to follow up on this request. Please let us know if we can be of further assistance or if your issue has been resolved.

---

<div class="post-metadata">

**Author:** ![ramit\_goolry](https://avatars.discourse-cdn.com/v4/letter/r/85e7bf/32.png) [@ramit\_goolry](https://community.wandb.ai/u/ramit_goolry)\
**Post date:** [April 10, 2023, 4:49pm UTC](https://community.wandb.ai/t/integrating-wandb-and-aws-lambda-multiprocessing/4160/4 "2023-04-10T16:49:28Z")

</div>

Hi @chiara_mc, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!

---

<div class="post-metadata">

**Author:** ![chiara\_mc](https://avatars.discourse-cdn.com/v4/letter/c/94ad74/32.png) [@chiara\_mc](https://community.wandb.ai/u/chiara_mc)\
**Post date:** [April 11, 2023, 2:14pm UTC](https://community.wandb.ai/t/integrating-wandb-and-aws-lambda-multiprocessing/4160/5 "2023-04-11T14:14:59Z")

</div>

Hey @ramit_goolry, thanks for your reply!

We need the lambda to be able to load models and run them, and we liked the idea of being able to retrieve them directly from our wandb’s model registry using aliases, such that we can easily change what models are being used in prod from the wandb website.

The workaround we are using is to store all model version in our own s3 bucket, using artifact references so that we can still manage the model versions through wandb. The lambda is then able to call wandb to know which versions to use (using the ETag), and can load them from S3.

Actually, we realised that while `artifact.download()` causes issues, we can download individual files using `artifact.get_path(model_path).download()` without issues (at least when they are stored on S3, not sure if it would also work if they were stored on wandb).

If you have any thoughts on this I’d be keen to hear them!

---

<div class="post-metadata">

**Author:** ![ramit\_goolry](https://avatars.discourse-cdn.com/v4/letter/r/85e7bf/32.png) [@ramit\_goolry](https://community.wandb.ai/u/ramit_goolry)\
**Post date:** [April 18, 2023, 5:54pm UTC](https://community.wandb.ai/t/integrating-wandb-and-aws-lambda-multiprocessing/4160/6 "2023-04-18T17:54:03Z")

</div>

Hey @chiara_mc,

I see. Since Semaphores are not available on AWS Lambda, Artifacts are not directly going to be available through Lambda. If you are using Artifact references linked to an S3 bucket, you should be able to use the W&B API to get the reference’s path and use that alongside boto3 to download them.

Thanks,  
Ramit

---

<div class="post-metadata">

**Author:** ![ramit\_goolry](https://avatars.discourse-cdn.com/v4/letter/r/85e7bf/32.png) [@ramit\_goolry](https://community.wandb.ai/u/ramit_goolry)\
**Post date:** [April 24, 2023, 4:02pm UTC](https://community.wandb.ai/t/integrating-wandb-and-aws-lambda-multiprocessing/4160/7 "2023-04-24T16:02:34Z")

</div>

Hi Chiara,

We wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.

Best,  
Weights & Biases

---

<div class="post-metadata">

**Author:** ![ramit\_goolry](https://avatars.discourse-cdn.com/v4/letter/r/85e7bf/32.png) [@ramit\_goolry](https://community.wandb.ai/u/ramit_goolry)\
**Post date:** [April 26, 2023, 7:19pm UTC](https://community.wandb.ai/t/integrating-wandb-and-aws-lambda-multiprocessing/4160/8 "2023-04-26T19:19:04Z")

</div>

Hi Chiara, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!

---

<div class="post-metadata">

**Author:** ![sevenmentor44](https://avatars.discourse-cdn.com/v4/letter/s/e68b1a/32.png) [@sevenmentor44](https://community.wandb.ai/u/sevenmentor44)\
**Post date:** [May 2, 2023, 10:28am UTC](https://community.wandb.ai/t/integrating-wandb-and-aws-lambda-multiprocessing/4160/9 "2023-05-02T10:28:20Z")

</div>

great post. thanks for sharing. AWS Training in Pune enables you to attain the skills needed to clear the AWS Certified Solutions Architect exam. Join now for the best AWS Certification Course in Pune.

[AWS course in Pune](https://www.sevenmentor.com/amazon-web-services-training-institute-in-pune.php)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/flex020/uploads/wandb/original/1X/366b649231631dbab896843020da0056074ac79d.png) [@system](https://community.wandb.ai/u/system)\
**Post date:** [July 1, 2023, 10:28am UTC](https://community.wandb.ai/t/integrating-wandb-and-aws-lambda-multiprocessing/4160/10 "2023-07-01T10:28:58Z")

</div>

Closed on July 1, 2023, 10:28am UTC

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.