Integration of Wandb with AWS Lambda

W&B Help
,
1

Hi Community,

I was trying to integrate the WandB inside the AWS lambda. The code was deployed successfully but during testing I am facing an issue with WandB.

OSError: [Errno 38] Function not implemented

This error is being raised by the WandB library inside the AWS lambda. This is probably because of multiprocessing utilized by WandB. Below are more details about the Lambda configuration and env variables:

CONFIGURATION:
Python Version= 3.7
Architecture = x86_64

ENV VARIABLES
WANDB_API_KEY = Key
WANDB_CACHE_DIR= /tmp/
WANDB_CONFIG_DIR=/tmp/
WANDB_DIR=/tmp/
WANDB_SILENT=true

Is there any suggestion how to deal with this bug?

2

Hey Akshey, can you send the full stack trace?

3

022-04-21T08:16:02.301-03:00

wandb: WARNING Path wandb/ wasn’t writable, using system temp directory.

2022-04-21T08:16:02.678-03:00

Problem at: /var/task/execution.py 103 lambda_handler_dummy

2022-04-21T08:16:02.682-03:00

Traceback (most recent call last):

2022-04-21T08:16:02.683-03:00

Copy
File “/var/task/wandb/sdk/wandb_init.py”, line 996, in init

2022-04-21T08:16:02.683-03:00

run = wi.init()

2022-04-21T08:16:02.683-03:00

File “/var/task/wandb/sdk/wandb_init.py”, line 532, in init

2022-04-21T08:16:02.683-03:00

backend.ensure_launched()

2022-04-21T08:16:02.683-03:00

File “/var/task/wandb/sdk/backend/backend.py”, line 188, in ensure_launched

2022-04-21T08:16:02.683-03:00

self.record_q = self._multiprocessing.Queue()

2022-04-21T08:16:02.683-03:00

File “/var/lang/lib/python3.7/multiprocessing/context.py”, line 102, in Queue

2022-04-21T08:16:02.683-03:00

return Queue(maxsize, ctx=self.get_context())

2022-04-21T08:16:02.683-03:00

File “/var/lang/lib/python3.7/multiprocessing/queues.py”, line 42, in init

2022-04-21T08:16:02.683-03:00

self._rlock = ctx.Lock()

2022-04-21T08:16:02.683-03:00

File “/var/lang/lib/python3.7/multiprocessing/context.py”, line 67, in Lock

2022-04-21T08:16:02.683-03:00

return Lock(ctx=self.get_context())

2022-04-21T08:16:02.683-03:00

File “/var/lang/lib/python3.7/multiprocessing/synchronize.py”, line 162, in init

2022-04-21T08:16:02.683-03:00

SemLock.init(self, SEMAPHORE, 1, 1, ctx=ctx)

2022-04-21T08:16:02.683-03:00

File “/var/lang/lib/python3.7/multiprocessing/synchronize.py”, line 59, in init

2022-04-21T08:16:02.683-03:00

unlink_now)

2022-04-21T08:16:02.683-03:00

OSError: [Errno 38] Function not implemented

4

@armanharutyunyan ,I experience the same

5

Hey @lexofir @akshey, we have made some changes that should fix this. Can you update your client and check if you still get the error?

6

Hi there, I wanted to follow up on this request. Please let us know if we can be of further assistance or if your issue has been resolved.

7

Hey there, since we have not heard back from you, I’ll be closing the ticket. But please ping me if you still experience this.

8

Hello @armanharutyunyan
I am having the same issue with using wandb within AWS Lambda ( which does not support python multiprocessing )

How do i resolve this? Is there a way to disable multiprocessing within wandb via an environment variable?

For fastai to work within a lambda, we just have to set num_workers=0 …is there a similar fix available here?

9

Im having the same issue

10

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.