Github authentication issue when launching run on VM

[pjsheehan]

Hi there! I’m trying to setup a launch agent in a GCP GCE VM. The agent is starting fine, but when I submit my first job, the agent attempts to clone the repository and github asks for a username/password. Seems like the agent is requesting over https instead of ssh (I have a key configured and can clone that way fine).

Any tips/tricks here? Here’s the logs:

12 02:00:16pm wandb:              'overrides': {'args': [],
13 02:00:16pm wandb:                            'entry_point': [],
14 02:00:16pm wandb:                            'run_config': {'site': 'texas-site'}},
15 02:00:16pm wandb:              'project': '2023 Net Load Forecasting Prize',
16 02:00:16pm wandb:              'resource': 'local-container',
17 02:00:16pm wandb:              'resource_args': {'local-container': {}}}}
18 02:00:16pm wandb:
19 02:00:16pm wandb: launch: Launching job: pjsheehan/2023 Net Load Forecasting Prize/job-git_github.com_patricksheehan_2023-Net-Load-Forecasting-Prize.git_scripts_production_submit_forecast_for_site.py:latest
20 02:00:16pm wandb:   2 of 2 files downloaded.
21 02:00:26pm wandb: launch: agent ywa2aqxo running 1 out of a maximum of 1 jobs
22 02:00:34pm wandb: ERROR launch: Error running job: Traceback (most recent call last):
23 02:00:34pm wandb: ERROR   File "/home/sheehanpj15/.local/lib/python3.9/site-packages/wandb/sdk/launch/agent/agent.py", line 461, in thread_run_job
24 02:00:34pm wandb: ERROR     self._thread_run_job(launch_spec, job, default_config, api, thread_id)
25 02:00:34pm wandb: ERROR   File "/home/sheehanpj15/.local/lib/python3.9/site-packages/wandb/sdk/launch/agent/agent.py", line 487, in _thread_run_job
26 02:00:34pm wandb: ERROR     project = fetch_and_validate_project(project, api)
27 02:00:34pm wandb: ERROR   File "/home/sheehanpj15/.local/lib/python3.9/site-packages/wandb/sdk/launch/_project_spec.py", line 488, in fetch_and_validate_project
28 02:00:34pm wandb: ERROR     launch_project._fetch_job()
29 02:00:34pm wandb: ERROR   File "/home/sheehanpj15/.local/lib/python3.9/site-packages/wandb/sdk/launch/_project_spec.py", line 271, in _fetch_job
30 02:00:34pm wandb: ERROR     job.configure_launch_project(self)
31 02:00:34pm wandb: ERROR   File "/home/sheehanpj15/.local/lib/python3.9/site-packages/wandb/apis/public.py", line 4357, in _configure_launch_project_repo
32 02:00:34pm wandb: ERROR     _fetch_git_repo(
33 02:00:34pm wandb: ERROR   File "/home/sheehanpj15/.local/lib/python3.9/site-packages/wandb/sdk/launch/utils.py", line 453, in _fetch_git_repo
34 02:00:34pm wandb: ERROR     ref.fetch(dst_dir)
35 02:00:34pm wandb: ERROR   File "/home/sheehanpj15/.local/lib/python3.9/site-packages/wandb/sdk/launch/github_reference.py", line 154, in fetch
36 02:00:34pm wandb: ERROR     origin.fetch(depth=1)
37 02:00:34pm wandb: ERROR   File "/home/sheehanpj15/.local/lib/python3.9/site-packages/git/remote.py", line 1015, in fetch
38 02:00:34pm wandb: ERROR     res = self._get_fetch_info_from_stderr(proc, progress, kill_after_timeout=kill_after_timeout)
39 02:00:34pm wandb: ERROR   File "/home/sheehanpj15/.local/lib/python3.9/site-packages/git/remote.py", line 854, in _get_fetch_info_from_stderr
40 02:00:34pm wandb: ERROR     proc.wait(stderr=stderr_text)
41 02:00:34pm wandb: ERROR   File "/home/sheehanpj15/.local/lib/python3.9/site-packages/git/cmd.py", line 604, in wait
42 02:00:34pm wandb: ERROR     raise GitCommandError(remove_password_if_present(self.args), status, errstr)
43 02:00:34pm wandb: ERROR git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
44 02:00:34pm wandb: ERROR   cmdline: git fetch -v --depth=1 -- origin
45 02:00:34pm wandb: ERROR   stderr: 'fatal: Authentication failed for 'https://github.com/patricksheehan/2023-Net-Load-Forecasting-Prize/''
46 02:00:34pm wandb: ERROR
47 02:00:36pm wandb: launch: Shutting down, active jobs:
48 02:00:36pm wandb: launch: agent ywa2aqxo polling on queues NLFC-2023, running 0 out of a maximum of 1 jobs

[pjsheehan]

FWIW a workaround seems to be to set up a personal access token for the VM like this: Caching your GitHub credentials in Git - GitHub Docs

In general, I’m sort of surprised there’s no tutorial provided by W&B for a really simple cloud agent setup via a repo:

• Start your VM
• Install W&B
• Install docker CLI
• Authenticate your repo
• Run the agent command
• Launch

The existing documentation made me think I needed to already have a docker container setup for my project, etc…

[nathank]

Hi @pjsheehan, glad you were able to get a resolution on this. I can raise this to our launch team and see if there is a working example of this that we can make public.

I can also mention this to the docs team to make it more clear that it is possible without a container and by either saving your code in W&B or using a git repo to pull your code.