Hi,
I am trying to set up launch agents using a custom Docker file and encountering a problem and am curious if others have encountered the same or have some recommendations.
For the scenario I am currently exploring I would like to set up launch agents without having to set up a local wandb installation on each agent host by using Docker (compose) instead.
I have built a custom image (and compose file) that sets up a working wandb installation. Since the agent needs access to Docker itself the image also includes the Docker CLI and it gets mounted the Docker socket of the host system (so I am not using the Docker-in-Docker-method [DinD] approach).
When launching a job using a base image and local code the agent will run “docker run” using the volume flag like this “–volume /tmp/tmpwll8n_n1:/mnt/wandb” where the temp subdirectory is dynamically generated. The problem seems to be that due to the agent using its hosts Docker socket that the left-hand-side of this volume expression now refers to a directory on the host system not one of the docker agent itself thus the /mnt/wandb will not include the code files and an entry-point like “python3 my-run.py” will fail. I see no way to control the name of genarated temp sub-directory.
Can anyone give pointers to an elegant way to solve this or does anyone can give their experience regarding using DinD instead of the socket approach in this scenario?