Wandb: ERROR Error while calling W&B API: internal error

W&B Help
1

Hi,

Using wandb for sweeps and it was working well until getting the following error:

wandb: ERROR Error while calling W&B API: An internal error occurred. Please contact support. (<Response [500]>)

I assumed that the server was down but checking status.wandb.ai suggested it is currently up. Any help would be appreciated.

Using version 0.16.4 and python 3.10.13

2

I also started having this error since a few hours ago.

Removing .netrc and relogin did not work. I also could not manually stop a run on the wandb.ai web. The runs all crashed.

last lines of the debug-internal.log is:

2024-03-30 22:07:54,576 INFO HandlerThread:30916 [handler.py:handle_request_defer():172] handle defer: 9
2024-03-30 22:07:55,018 DEBUG HandlerThread:30916 [handler.py:handle_request():146] handle_request: poll_exit
2024-03-30 22:07:55,394 INFO Thread-17 :30916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /Users/xxxxx/xxxx/wandb/run-20240330_220531-xxxxx/files/output.log
2024-03-30 22:07:55,571 ERROR SenderThread:30916 [internal_api.py:execute():373] 500 response executing GraphQL.
2024-03-30 22:07:55,571 ERROR SenderThread:30916 [internal_api.py:execute():374] {“errors”:[{“message”:“An internal error occurred. Please contact support.”,“path”:[“createArtifactManifest”]}],“data”:{“createArtifactManifest”:null}}

3

I’m experiencing the same issue since an hour ago.

4

Just did another run and it completed successfully.

1 Like
5

I’m experiencing the same issue

6

Me too. Also same error.
It starts OK logs a few batches but then starts showing up
wandb: ERROR Error while calling W&B API: An internal error occurred. Please contact support. (<Response [500]>)

7

Hi @speatjack , @bo-xu14 , @fre-lax , @dinu23 , @portavales sorry for running into this and for flagging this behavior. I got an update from our team that this should be working now, kindly confirm if that is the case on your end guys, thanks~

8

Dear Jack,

Thanks for reaching out to us and for highlighting this and sincerely apologize for the inconvenience. I wonder if you would object to advising if this is still occurring for you, we had a server-side issue that was fixed over the weekend – and active runs should now be sending data to wandb without issue.

If you are still running into issues then we would be extremely grateful if you would be able to provide the debug.log and internal-debug.log associated with the affected run.

Look forward to hearing back from you.

Best wishes,

Frida
EMEA Support Manager

9

Hi Frida,

I’m happy to report that all is working now! Thank you for the response.

Best,
Jack

10

Dear Jack,

Thanks for the confirmation here!

I will go ahead and close this off for you but please don’t hesitate to reach out on this or anything else.

Best wishes,

Frida