This one line causes tons of errors in debug-internal.log
wandb.run.log_code("/fly") # /fly is the working dir in my docker container
2024-01-12 03:06:14,062 DEBUG HandlerThread:1657 [handler.py:handle_request():140] handle_request: log_artifact
2024-01-12 03:06:14,472 DEBUG SenderThread:1657 [sender.py:send():236] send: telemetry
2024-01-12 03:06:14,473 DEBUG SenderThread:1657 [sender.py:send():236] send: config
2024-01-12 03:06:14,475 DEBUG SenderThread:1657 [sender.py:send():236] send: metric
2024-01-12 03:06:14,475 DEBUG SenderThread:1657 [sender.py:send():236] send: telemetry
2024-01-12 03:06:14,475 DEBUG SenderThread:1657 [sender.py:send():236] send: metric
2024-01-12 03:06:14,475 WARNING SenderThread:1657 [sender.py:send_metric():908] Seen metric with glob (shouldn't happen)
2024-01-12 03:06:14,475 DEBUG SenderThread:1657 [sender.py:send():236] send: telemetry
2024-01-12 03:06:14,476 DEBUG SenderThread:1657 [sender.py:send_request():250] send_request: log_artifact
2024-01-12 03:06:14,485 INFO Thread-7 :1657 [dir_watcher.py:_on_file_created():269] file/dir created: /fly/wandb/run-20240112_030612-3vmc4gwo/files/output.log
2024-01-12 03:06:14,486 INFO Thread-7 :1657 [dir_watcher.py:_on_file_created():269] file/dir created: /fly/wandb/run-20240112_030612-3vmc4gwo/files/code/run_glue.py
2024-01-12 03:06:14,486 INFO Thread-7 :1657 [dir_watcher.py:_on_file_created():269] file/dir created: /fly/wandb/run-20240112_030612-3vmc4gwo/files/diff.patch
2024-01-12 03:06:14,486 INFO Thread-7 :1657 [dir_watcher.py:_on_file_created():269] file/dir created: /fly/wandb/run-20240112_030612-3vmc4gwo/files/wandb-metadata.json
2024-01-12 03:06:14,486 INFO Thread-7 :1657 [dir_watcher.py:_on_file_created():269] file/dir created: /fly/wandb/run-20240112_030612-3vmc4gwo/files/requirements.txt
2024-01-12 03:06:14,486 INFO Thread-7 :1657 [dir_watcher.py:_on_file_created():269] file/dir created: /fly/wandb/run-20240112_030612-3vmc4gwo/files/wandb-summary.json
2024-01-12 03:06:14,487 INFO Thread-7 :1657 [dir_watcher.py:_on_file_created():269] file/dir created: /fly/wandb/run-20240112_030612-3vmc4gwo/files/code
2024-01-12 03:06:14,579 INFO Thread-11 :1657 [upload_job.py:push():138] Uploaded file /tmp/tmpj5qog504wandb/1ztzcv15-wandb-metadata.json
2024-01-12 03:06:14,846 INFO Thread-12 :1657 [upload_job.py:push():138] Uploaded file /tmp/tmpj5qog504wandb/x215prhl-code/run_glue.py
2024-01-12 03:06:14,978 INFO Thread-13 :1657 [upload_job.py:push():138] Uploaded file /tmp/tmpj5qog504wandb/3cirfn6a-diff.patch
2024-01-12 03:06:16,484 INFO Thread-7 :1657 [dir_watcher.py:_on_file_modified():286] file/dir modified: /fly/wandb/run-20240112_030612-3vmc4gwo/files/output.log
2024-01-12 03:06:18,485 INFO Thread-7 :1657 [dir_watcher.py:_on_file_modified():286] file/dir modified: /fly/wandb/run-20240112_030612-3vmc4gwo/files/output.log
2024-01-12 03:06:20,486 INFO Thread-7 :1657 [dir_watcher.py:_on_file_modified():286] file/dir modified: /fly/wandb/run-20240112_030612-3vmc4gwo/files/output.log
2024-01-12 03:06:22,487 INFO Thread-7 :1657 [dir_watcher.py:_on_file_modified():286] file/dir modified: /fly/wandb/run-20240112_030612-3vmc4gwo/files/output.log
2024-01-12 03:06:24,487 INFO Thread-7 :1657 [dir_watcher.py:_on_file_modified():286] file/dir modified: /fly/wandb/run-20240112_030612-3vmc4gwo/files/output.log
2024-01-12 03:06:24,502 DEBUG HandlerThread:1657 [handler.py:handle_request():140] handle_request: partial_history
2024-01-12 03:06:25,701 INFO Thread-74 :1657 [upload_job.py:push():96] Uploaded file /home/user/.cache/wandb/artifacts/obj/md5/c4/b3de2b9c3bc97e095825d7779b7c4a
2024-01-12 03:06:26,489 INFO Thread-7 :1657 [dir_watcher.py:_on_file_modified():286] file/dir modified: /fly/wandb/run-20240112_030612-3vmc4gwo/files/output.log
2024-01-12 03:06:28,178 ERROR Thread-24 :1657 [internal_api.py:upload_file():1694] upload_file exception https://storage.googleapis.com/wandb-artifacts-prod/wandb_artifacts/129557231/688689262/f6d8e0f2d358f8b7efef87460bd58500?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=pTF29imMaH4Eb%2BOIbSOyUePEq0aQnFm7RKxwWzJBxjwnnbpg08yxkXQ4ewS%2BZbcm2dWFasd%2BazcJUfC5l43%2FVgG0kKeOc%2BXCRhN0kEQdPcCFm2xO4GR1cRQuwk3Y1gDOJdst94ZumKo27dqHbhZO9aMBOiHEc4UxrrCvMyp3mzHmkkhE%2Fo4H6UuNVUlikbNXY8YU1VfmQY8m4PIMonpQfwFE%2F0%2FrMdKl%2FspGJ4813aCA0X2WJcUQW3DnqZF4CQaGaEgGr%2Br1S3vEVfPBggZJfpiW2M13p%2FPPCIoWM6g%2BKwB%2BzqY5EVgtxBSWqC0oupO%2BrWtyas%2FAVqXFhPW08CIPZg%3D%3D: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /wandb-artifacts-prod/wandb_artifacts/129557231/688689262/f6d8e0f2d358f8b7efef87460bd58500?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=pTF29imMaH4Eb%2BOIbSOyUePEq0aQnFm7RKxwWzJBxjwnnbpg08yxkXQ4ewS%2BZbcm2dWFasd%2BazcJUfC5l43%2FVgG0kKeOc%2BXCRhN0kEQdPcCFm2xO4GR1cRQuwk3Y1gDOJdst94ZumKo27dqHbhZO9aMBOiHEc4UxrrCvMyp3mzHmkkhE%2Fo4H6UuNVUlikbNXY8YU1VfmQY8m4PIMonpQfwFE%2F0%2FrMdKl%2FspGJ4813aCA0X2WJcUQW3DnqZF4CQaGaEgGr%2Br1S3vEVfPBggZJfpiW2M13p%2FPPCIoWM6g%2BKwB%2BzqY5EVgtxBSWqC0oupO%2BrWtyas%2FAVqXFhPW08CIPZg%3D%3D (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))
2024-01-12 03:06:28,178 ERROR Thread-24 :1657 [internal_api.py:upload_file():1696] upload_file request headers: {'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-MD5': '9tjg8tNY+Lfv74dGC9WFAA==', 'Content-Type': 'text/x-python; charset=utf-8', 'Content-Length': '3117'}
2024-01-12 03:06:28,178 ERROR Thread-24 :1657 [internal_api.py:upload_file():1698] upload_file response body:
2024-01-12 03:06:28,490 INFO Thread-7 :1657 [dir_watcher.py:_on_file_modified():286] file/dir modified: /fly/wandb/run-20240112_030612-3vmc4gwo/files/output.log
2024-01-12 03:06:29,479 DEBUG HandlerThread:1657 [handler.py:handle_request():140] handle_request: stop_status
2024-01-12 03:06:30,038 ERROR Thread-62 :1657 [internal_api.py:upload_file():1694] upload_file exception https://storage.googleapis.com/wandb-artifacts-prod/wandb_artifacts/129557231/688689262/a24a1e5a97a2c8e5d911e4bfae536f72?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=eDFpNPhMjf9RK4GacboWc0DW7Zbo%2BOHJrS9plJO%2BgDiw%2BJmM4FKl7JzbrwHQora7HhTwBJF4u3UMUdr1jFbvoDMQkWbwFFoXKpOXg1JtQBVjxl5%2BrS102lurDOY8yyiek%2BzbzpqZppM7l1npSK2vPYArNtOYdgqTjEW7ymOIf4KgVOTNsnTOe1OQPQJALHvKHbEfWDbA7%2BE%2FJjdPe1lBZ9WLqo45zP%2BbzVW1lQ1%2Fhb%2B0p4rt0gbhKPEVvq38Vq1j6JXnI5vTzM21Wu%2F5hfgnpAcHiwg69UmQo%2FqmECl1is8w6wNzi8MF1C%2Bnmb0VeGjGn5rrUB2cvV0mRA4zBmmoWw%3D%3D: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /wandb-artifacts-prod/wandb_artifacts/129557231/688689262/a24a1e5a97a2c8e5d911e4bfae536f72?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=eDFpNPhMjf9RK4GacboWc0DW7Zbo%2BOHJrS9plJO%2BgDiw%2BJmM4FKl7JzbrwHQora7HhTwBJF4u3UMUdr1jFbvoDMQkWbwFFoXKpOXg1JtQBVjxl5%2BrS102lurDOY8yyiek%2BzbzpqZppM7l1npSK2vPYArNtOYdgqTjEW7ymOIf4KgVOTNsnTOe1OQPQJALHvKHbEfWDbA7%2BE%2FJjdPe1lBZ9WLqo45zP%2BbzVW1lQ1%2Fhb%2B0p4rt0gbhKPEVvq38Vq1j6JXnI5vTzM21Wu%2F5hfgnpAcHiwg69UmQo%2FqmECl1is8w6wNzi8MF1C%2Bnmb0VeGjGn5rrUB2cvV0mRA4zBmmoWw%3D%3D (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))
2024-01-12 03:06:30,039 ERROR Thread-62 :1657 [internal_api.py:upload_file():1696] upload_file request headers: {'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-MD5': 'okoeWpeiyOXZEeS/rlNvcg==', 'Content-Type': 'text/x-python; charset=utf-8', 'Content-Length': '10246'}
2024-01-12 03:06:30,039 ERROR Thread-62 :1657 [internal_api.py:upload_file():1698] upload_file response body:
2024-01-12 03:06:30,097 ERROR Thread-71 :1657 [internal_api.py:upload_file():1694] upload_file exception https://storage.googleapis.com/wandb-artifacts-prod/wandb_artifacts/129557231/688689262/a4ef7307b21b62ee3a03ca9fabfc8b06?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=Zupo0IMJHnQaFqTuu%2Fx9JRC7zLFfEIPGzQrst6Q0XzxnvO9nMy7xX%2Bkx0PHv0onqKNcGc4N%2FKg4gNkVEt7w7XTc9jmjhJwJ5xUovCZ2qmHSs9WCnPh%2FEjBfdB4xKuyMoG53eSd%2F3LyOJy2K78jMokyM5gUaP3mL7HQCVuPVm7n1YXEcjAKg%2FkvtVtrTG0VrBQ0QAy2lu5fk%2F3StCfMLFJzVN7RFjm9TkatxCIyU4oWwl9aNE0hNTTVVCWdxMr3DHLA5k1CV%2B%2FuuHdSn5kpijSQe2IzXnUtICO6XIS1W7Ij4BqutZI8x8RkH2K7e1aRtuTXZli7g%2Fx%2BiNkM%2FBpiWUfg%3D%3D: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /wandb-artifacts-prod/wandb_artifacts/129557231/688689262/a4ef7307b21b62ee3a03ca9fabfc8b06?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=Zupo0IMJHnQaFqTuu%2Fx9JRC7zLFfEIPGzQrst6Q0XzxnvO9nMy7xX%2Bkx0PHv0onqKNcGc4N%2FKg4gNkVEt7w7XTc9jmjhJwJ5xUovCZ2qmHSs9WCnPh%2FEjBfdB4xKuyMoG53eSd%2F3LyOJy2K78jMokyM5gUaP3mL7HQCVuPVm7n1YXEcjAKg%2FkvtVtrTG0VrBQ0QAy2lu5fk%2F3StCfMLFJzVN7RFjm9TkatxCIyU4oWwl9aNE0hNTTVVCWdxMr3DHLA5k1CV%2B%2FuuHdSn5kpijSQe2IzXnUtICO6XIS1W7Ij4BqutZI8x8RkH2K7e1aRtuTXZli7g%2Fx%2BiNkM%2FBpiWUfg%3D%3D (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))
2024-01-12 03:06:30,097 ERROR Thread-71 :1657 [internal_api.py:upload_file():1696] upload_file request headers: {'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-MD5': 'pO9zB7IbYu46A8qfq/yLBg==', 'Content-Type': 'text/x-python; charset=utf-8', 'Content-Length': '5107'}
2024-01-12 03:06:30,097 ERROR Thread-71 :1657 [internal_api.py:upload_file():1698] upload_file response body:
2024-01-12 03:06:30,319 ERROR Thread-50 :1657 [internal_api.py:upload_file():1694] upload_file exception https://storage.googleapis.com/wandb-artifacts-prod/wandb_artifacts/129557231/688689262/3406fd5e468d182e5784140b8180567c?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=fr2Q7kLu3ror3WC6nwb%2FVyNOaz5QBpq28bu5b0NZwd1smY6chVQ0qR2XR803rUD%2FxkDuHGljur9S%2Fe7HdY%2FXD5ZiUJhyx4CAgfAD2BOhpLnbnEtNM2Thy6gpAxJAW%2B7IbFPoaCv%2B5pJCbCprNTD4UFuoWypaw8C4jfguCUe1AYaIDiooNhvqTNjn7brlqfF730tEdyvv4pdoSdphxdfuqrvWfsjoQ1fph5gHU7DF8jLR8LQ3BiCrB8eCpgU1%2FPYSAHefmddqA0wZypDGarH3PiJWsyR7ZoJNjVyovyXpfp0BAZBqsF3vN%2BwH3hoGKHQTi2VsLHNnf0%2BekCqjUNMmwA%3D%3D: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /wandb-artifacts-prod/wandb_artifacts/129557231/688689262/3406fd5e468d182e5784140b8180567c?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=fr2Q7kLu3ror3WC6nwb%2FVyNOaz5QBpq28bu5b0NZwd1smY6chVQ0qR2XR803rUD%2FxkDuHGljur9S%2Fe7HdY%2FXD5ZiUJhyx4CAgfAD2BOhpLnbnEtNM2Thy6gpAxJAW%2B7IbFPoaCv%2B5pJCbCprNTD4UFuoWypaw8C4jfguCUe1AYaIDiooNhvqTNjn7brlqfF730tEdyvv4pdoSdphxdfuqrvWfsjoQ1fph5gHU7DF8jLR8LQ3BiCrB8eCpgU1%2FPYSAHefmddqA0wZypDGarH3PiJWsyR7ZoJNjVyovyXpfp0BAZBqsF3vN%2BwH3hoGKHQTi2VsLHNnf0%2BekCqjUNMmwA%3D%3D (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))
2024-01-12 03:06:30,319 ERROR Thread-50 :1657 [internal_api.py:upload_file():1696] upload_file request headers: {'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-MD5': 'NAb9XkaNGC5XhBQLgYBWfA==', 'Content-Type': 'text/x-python; charset=utf-8', 'Content-Length': '905'}
2024-01-12 03:06:30,319 ERROR Thread-50 :1657 [internal_api.py:upload_file():1698] upload_file response body:
2024-01-12 03:06:30,414 ERROR Thread-58 :1657 [internal_api.py:upload_file():1694] upload_file exception https://storage.googleapis.com/wandb-artifacts-prod/wandb_artifacts/129557231/688689262/5dbac378cb160a651a8fdb255b071a65?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=AfnTxwHwmuWgHPKRQzEYH8DkRMUVvo2BCX3rN1JVxGrr3EtoRLvln5vn3VPGMArEq7f3Z7%2BnkhTLNZWBR9Tbq5uHLj07StjMwwCoI8IMVHaIKaWracOGR0K55usXenAZmhl2SMoNCsNJHHgJNeAB2%2BcktpOPIKWnpkNzL8Zf5vtx%2FgSk5fITTdiAgFZr364MqjWbWwMxSvii0lC6bVHXsdTxZcYr0JputmHB23k6cNmzmV%2FEvrfbBZ4x2TOoTLz31ouVKSGlYcfdbTogFbdVlhC8aBsjh5tWKYSZfBhuHtN4fnOlbx9obBTAWZJ7eN5ZrvAVgyDhG99BSZ9gaY1uQA%3D%3D: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /wandb-artifacts-prod/wandb_artifacts/129557231/688689262/5dbac378cb160a651a8fdb255b071a65?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=AfnTxwHwmuWgHPKRQzEYH8DkRMUVvo2BCX3rN1JVxGrr3EtoRLvln5vn3VPGMArEq7f3Z7%2BnkhTLNZWBR9Tbq5uHLj07StjMwwCoI8IMVHaIKaWracOGR0K55usXenAZmhl2SMoNCsNJHHgJNeAB2%2BcktpOPIKWnpkNzL8Zf5vtx%2FgSk5fITTdiAgFZr364MqjWbWwMxSvii0lC6bVHXsdTxZcYr0JputmHB23k6cNmzmV%2FEvrfbBZ4x2TOoTLz31ouVKSGlYcfdbTogFbdVlhC8aBsjh5tWKYSZfBhuHtN4fnOlbx9obBTAWZJ7eN5ZrvAVgyDhG99BSZ9gaY1uQA%3D%3D (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))
2024-01-12 03:06:30,414 ERROR Thread-58 :1657 [internal_api.py:upload_file():1696] upload_file request headers: {'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-MD5': 'XbrDeMsWCmUaj9slWwcaZQ==', 'Content-Type': 'text/x-python; charset=utf-8', 'Content-Length': '2985'}
2024-01-12 03:06:30,414 ERROR Thread-58 :1657 [internal_api.py:upload_file():1698] upload_file response body:
2024-01-12 03:06:30,426 ERROR Thread-54 :1657 [internal_api.py:upload_file():1694] upload_file exception https://storage.googleapis.com/wandb-artifacts-prod/wandb_artifacts/129557231/688689262/f14f5a4a6048e9a9df66034dff197b46?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=GU8XCzLvJHQ968QN7scgtmh0tlM%2B0GOW20UWSpuGx%2FQ5739RPe%2Bom4MF3PlHct4PSO1PN%2BP81ed8tO1m6sQ1NE1YwlXIqkICJR7T2pdL8QWpJyj0JImsLXcM9bel7gd5EtIpdq9j2oNA9lZ9gMKl3OWPkXb%2FkFWaq%2BO2lEW74eKTniP%2BfEnlXnw2gfLHVljABpvgAc6HD2I4lSF2x0qRGK%2FqjI9u3yYg%2BvWcNptr0%2Ft4MrXoo8HvjRtqfV%2BLZC8KUk72D12P4ao9nD08zylvZol1PKwVRnUOGczBBQsw0%2FstdfrFs7aVTDnHPf2qs00xGhV9i5L0mMyElV08yimoZw%3D%3D: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /wandb-artifacts-prod/wandb_artifacts/129557231/688689262/f14f5a4a6048e9a9df66034dff197b46?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=GU8XCzLvJHQ968QN7scgtmh0tlM%2B0GOW20UWSpuGx%2FQ5739RPe%2Bom4MF3PlHct4PSO1PN%2BP81ed8tO1m6sQ1NE1YwlXIqkICJR7T2pdL8QWpJyj0JImsLXcM9bel7gd5EtIpdq9j2oNA9lZ9gMKl3OWPkXb%2FkFWaq%2BO2lEW74eKTniP%2BfEnlXnw2gfLHVljABpvgAc6HD2I4lSF2x0qRGK%2FqjI9u3yYg%2BvWcNptr0%2Ft4MrXoo8HvjRtqfV%2BLZC8KUk72D12P4ao9nD08zylvZol1PKwVRnUOGczBBQsw0%2FstdfrFs7aVTDnHPf2qs00xGhV9i5L0mMyElV08yimoZw%3D%3D (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))
2024-01-12 03:06:30,426 ERROR Thread-54 :1657 [internal_api.py:upload_file():1696] upload_file request headers: {'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-MD5': '8U9aSmBI6anfZgNN/xl7Rg==', 'Content-Type': 'text/x-python; charset=utf-8', 'Content-Length': '34697'}
2024-01-12 03:06:30,426 ERROR Thread-54 :1657 [internal_api.py:upload_file():1698] upload_file response body:
2024-01-12 03:06:30,451 ERROR Thread-61 :1657 [internal_api.py:upload_file():1694] upload_file exception https://storage.googleapis.com/wandb-artifacts-prod/wandb_artifacts/129557231/688689262/0b7dc2a6d974854add70660ba55b3213?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=rxRouSB8fVPcHyCq%2B3eLZHd8KrhVKDTcCJC2NWkwrLvkNfDBe1HZFRKByfww2A49hjd7fi%2FMLQJ%2FFl4qcDcw9Z3U2igvZb5XIOxscf%2FYzJv%2BebZhVkuTB0jHhSEkB8MkYGeD9J1SqbndSVReE9cGEE6WS3wH5IleIcjEIdBOOToMpPGaGOh3%2FYYtkiJpqZYWDrzDEaQVSPNw5mk8ip%2FuikCqXj7cvsYUHPjthGf2rFI0TzE7EVGcKGgm3T2pyXCaabc4TZzuR4Jx%2BFFT6G%2FHubmZov9447UE1lb%2FiL98co430sh2uRXE1mqmvwCD87ji2J%2BvVVyrQr5S2yJveTPLqQ%3D%3D: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /wandb-artifacts-prod/wandb_artifacts/129557231/688689262/0b7dc2a6d974854add70660ba55b3213?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=rxRouSB8fVPcHyCq%2B3eLZHd8KrhVKDTcCJC2NWkwrLvkNfDBe1HZFRKByfww2A49hjd7fi%2FMLQJ%2FFl4qcDcw9Z3U2igvZb5XIOxscf%2FYzJv%2BebZhVkuTB0jHhSEkB8MkYGeD9J1SqbndSVReE9cGEE6WS3wH5IleIcjEIdBOOToMpPGaGOh3%2FYYtkiJpqZYWDrzDEaQVSPNw5mk8ip%2FuikCqXj7cvsYUHPjthGf2rFI0TzE7EVGcKGgm3T2pyXCaabc4TZzuR4Jx%2BFFT6G%2FHubmZov9447UE1lb%2FiL98co430sh2uRXE1mqmvwCD87ji2J%2BvVVyrQr5S2yJveTPLqQ%3D%3D (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))
2024-01-12 03:06:30,451 ERROR Thread-61 :1657 [internal_api.py:upload_file():1696] upload_file request headers: {'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-MD5': 'C33Cptl0hUrdcGYLpVsyEw==', 'Content-Type': 'text/x-python; charset=utf-8', 'Content-Length': '2620'}
2024-01-12 03:06:30,452 ERROR Thread-61 :1657 [internal_api.py:upload_file():1698] upload_file response body:
2024-01-12 03:06:30,477 ERROR Thread-64 :1657 [internal_api.py:upload_file():1694] upload_file exception https://storage.googleapis.com/wandb-artifacts-prod/wandb_artifacts/129557231/688689262/6ca56c4e367a9000b0c864fdb587a062?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=t9EoG1OHt9pYv0PEXrRj4WddBygFHehkOsuAslZy7ur%2F03BnTEuuxi67AWPO3j1KZf2FDNYvWQ7rCPV3MfcaGms2p81G6FH2MFgsDlqBt5ER3b65uceOJvbcfkiX%2Fr%2Fw7JwDKVSKgTIiL0CnVSvHYA6WHukl4o8B%2F3hRVMZjKTdzgp%2FLvvkAeUfOdKOdoIh0Bpd6PSCiOOKe79gqvcHz1JKCTcwZT%2B11m9cDGj73HxfbA7ZzjqrR7B2k54fn3Nl4T4LdzXVMWglh3zI8FwmJxdWuGy1FrnfpMlqGy3s7BdFMraOp6nGM25XVU7up57k8gqXj5OuqF43LL7Mlm1OHBQ%3D%3D: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /wandb-artifacts-prod/wandb_artifacts/129557231/688689262/6ca56c4e367a9000b0c864fdb587a062?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=t9EoG1OHt9pYv0PEXrRj4WddBygFHehkOsuAslZy7ur%2F03BnTEuuxi67AWPO3j1KZf2FDNYvWQ7rCPV3MfcaGms2p81G6FH2MFgsDlqBt5ER3b65uceOJvbcfkiX%2Fr%2Fw7JwDKVSKgTIiL0CnVSvHYA6WHukl4o8B%2F3hRVMZjKTdzgp%2FLvvkAeUfOdKOdoIh0Bpd6PSCiOOKe79gqvcHz1JKCTcwZT%2B11m9cDGj73HxfbA7ZzjqrR7B2k54fn3Nl4T4LdzXVMWglh3zI8FwmJxdWuGy1FrnfpMlqGy3s7BdFMraOp6nGM25XVU7up57k8gqXj5OuqF43LL7Mlm1OHBQ%3D%3D (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))
2024-01-12 03:06:30,477 ERROR Thread-64 :1657 [internal_api.py:upload_file():1696] upload_file request headers: {'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-MD5': 'bKVsTjZ6kACwyGT9tYegYg==', 'Content-Type': 'text/x-python; charset=utf-8', 'Content-Length': '6818'}
2024-01-12 03:06:30,478 ERROR Thread-64 :1657 [internal_api.py:upload_file():1698] upload_file response body:
2024-01-12 03:06:30,491 INFO Thread-7 :1657 [dir_watcher.py:_on_file_modified():286] file/dir modified: /fly/wandb/run-20240112_030612-3vmc4gwo/files/output.log
2024-01-12 03:06:30,581 ERROR Thread-59 :1657 [internal_api.py:upload_file():1694] upload_file exception https://storage.googleapis.com/wandb-artifacts-prod/wandb_artifacts/129557231/688689262/09ac9863f311c2f33d19d38309beb1fe?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=Q9YM87WtUfCqckhT%2B%2FRAXkHBEnszy%2B7P9QDoMOnaW8qpR6PfHCgane9WbdJsiCxSoHfJYJMPZ0yloL5hfv3EsZHQ%2BzRsUNOqj4pP%2FVmumW9PKHprYLFHGz1Wj7XsKT4065ShEuH8CDhSynUFZUHrrLj0lgq5UvQ2DwXUrSp%2BBZG5lo0urmBDU83PEpFV2kivdvRKqkZ8NFYxQuL2G9kjFsCS8MICBMMGd10Hkn7I6TiMzGX6yC%2FAJZvPAwjOXTQ%2B4EECfi0%2B0%2FJgB0%2BrI5aYk%2BPVPTvvvMB34Jcc0IOI9pfaivq1Maq5GWuCzmapyNTg6E16rGZkFKwpNj1qgwOwGQ%3D%3D: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /wandb-artifacts-prod/wandb_artifacts/129557231/688689262/09ac9863f311c2f33d19d38309beb1fe?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=Q9YM87WtUfCqckhT%2B%2FRAXkHBEnszy%2B7P9QDoMOnaW8qpR6PfHCgane9WbdJsiCxSoHfJYJMPZ0yloL5hfv3EsZHQ%2BzRsUNOqj4pP%2FVmumW9PKHprYLFHGz1Wj7XsKT4065ShEuH8CDhSynUFZUHrrLj0lgq5UvQ2DwXUrSp%2BBZG5lo0urmBDU83PEpFV2kivdvRKqkZ8NFYxQuL2G9kjFsCS8MICBMMGd10Hkn7I6TiMzGX6yC%2FAJZvPAwjOXTQ%2B4EECfi0%2B0%2FJgB0%2BrI5aYk%2BPVPTvvvMB34Jcc0IOI9pfaivq1Maq5GWuCzmapyNTg6E16rGZkFKwpNj1qgwOwGQ%3D%3D (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))
2024-01-12 03:06:30,581 ERROR Thread-59 :1657 [internal_api.py:upload_file():1696] upload_file request headers: {'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-MD5': 'CayYY/MRwvM9GdODCb6x/g==', 'Content-Type': 'text/x-python; charset=utf-8', 'Content-Length': '4041'}
2024-01-12 03:06:30,581 ERROR Thread-59 :1657 [internal_api.py:upload_file():1698] upload_file response body:
2024-01-12 03:06:30,615 ERROR Thread-20 :1657 [internal_api.py:upload_file():1694] upload_file exception https://storage.googleapis.com/wandb-artifacts-prod/wandb_artifacts/129557231/688689262/cc4032dc0054c69023f03483ca74a9be?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=DBiBXrRtqNKtkV10Lp4owHto6VBlTOLPclSq%2BH7gR754teIPIE6o%2BmNVa%2F7rO9peAWpW5jIfMTlY8YiX%2BwPAEOYZokRXhTmPXaZb%2FPKKf%2BVhsB%2FYH4pEqBllJ9nBDd9I6oPJNfYH9PRTd80krFlS4vUgIv98qgMCrly7RgtiknJ5CiAeDveOTM65Tk%2BUBCirCuMW61gk1MKwIimQ9YG4OayV0roDBxZWbVeQDqPBIJyFVfihAqYotP1D7VlJF7AzgzN12XND5Q5LS7uJXAZ20hT0ZYRZsICO90%2B%2BsuDiMZhGVeBJ30bURdLwSIAVTwWXJxbIZQF95Wq82M9dfpkHJQ%3D%3D: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /wandb-artifacts-prod/wandb_artifacts/129557231/688689262/cc4032dc0054c69023f03483ca74a9be?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=DBiBXrRtqNKtkV10Lp4owHto6VBlTOLPclSq%2BH7gR754teIPIE6o%2BmNVa%2F7rO9peAWpW5jIfMTlY8YiX%2BwPAEOYZokRXhTmPXaZb%2FPKKf%2BVhsB%2FYH4pEqBllJ9nBDd9I6oPJNfYH9PRTd80krFlS4vUgIv98qgMCrly7RgtiknJ5CiAeDveOTM65Tk%2BUBCirCuMW61gk1MKwIimQ9YG4OayV0roDBxZWbVeQDqPBIJyFVfihAqYotP1D7VlJF7AzgzN12XND5Q5LS7uJXAZ20hT0ZYRZsICO90%2B%2BsuDiMZhGVeBJ30bURdLwSIAVTwWXJxbIZQF95Wq82M9dfpkHJQ%3D%3D (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))
2024-01-12 03:06:30,615 ERROR Thread-20 :1657 [internal_api.py:upload_file():1696] upload_file request headers: {'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-MD5': 'zEAy3ABUxpAj8DSDynSpvg==', 'Content-Type': 'text/x-python; charset=utf-8', 'Content-Length': '110'}
2024-01-12 03:06:30,615 ERROR Thread-20 :1657 [internal_api.py:upload_file():1698] upload_file response body:
2024-01-12 03:06:30,633 ERROR Thread-78 :1657 [internal_api.py:upload_file():1694] upload_file exception https://storage.googleapis.com/wandb-artifacts-prod/wandb_artifacts/129557231/688689262/a027564d1f2cfae87b7086532b7243b6?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=tZpZdfsWar86PkZoEbXiqwr6py%2FFaMPtFF%2BbGKfP%2FM7XkYQuA5ycHtgTZYxxKahJK9eSgxZ97v%2FOW2afvPjf%2FCyXJwvw2K5FOrLc0OooUDIWsOjEtjEHliD3wZAtSFBxJYbuhN5MiygESJy714bzdE8rPZPjsptUSxmVTm%2FKmsESSbdflVHto0N9p6f4%2FdCgEPTYGSrg46wPOK4rFbkNLdXV5JN%2FRxo%2ByzN51fl2niW1fF6zovCsOkLZtTVMGoBs0%2BEvIfPP7YQpNsPHtgo9jLY2wpSz%2BgQ44eZp1RkxjvWUPNBqTYtRxQCB8RIdSpKnwrlvPvUgqaGw0we6HmMF6Q%3D%3D: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /wandb-artifacts-prod/wandb_artifacts/129557231/688689262/a027564d1f2cfae87b7086532b7243b6?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=tZpZdfsWar86PkZoEbXiqwr6py%2FFaMPtFF%2BbGKfP%2FM7XkYQuA5ycHtgTZYxxKahJK9eSgxZ97v%2FOW2afvPjf%2FCyXJwvw2K5FOrLc0OooUDIWsOjEtjEHliD3wZAtSFBxJYbuhN5MiygESJy714bzdE8rPZPjsptUSxmVTm%2FKmsESSbdflVHto0N9p6f4%2FdCgEPTYGSrg46wPOK4rFbkNLdXV5JN%2FRxo%2ByzN51fl2niW1fF6zovCsOkLZtTVMGoBs0%2BEvIfPP7YQpNsPHtgo9jLY2wpSz%2BgQ44eZp1RkxjvWUPNBqTYtRxQCB8RIdSpKnwrlvPvUgqaGw0we6HmMF6Q%3D%3D (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))
2024-01-12 03:06:30,633 ERROR Thread-78 :1657 [internal_api.py:upload_file():1696] upload_file request headers: {'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-MD5': 'oCdWTR8s+uh7cIZTK3JDtg==', 'Content-Type': 'text/x-python; charset=utf-8', 'Content-Length': '2341'}
2024-01-12 03:06:30,634 ERROR Thread-78 :1657 [internal_api.py:upload_file():1698] upload_file response body:
2024-01-12 03:06:30,683 ERROR Thread-77 :1657 [internal_api.py:upload_file():1694] upload_file exception https://storage.googleapis.com/wandb-artifacts-prod/wandb_artifacts/129557231/688689262/a84ad5dd9242be0bd32175ceeb27ea74?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=Lo5cNnJP7Dk882JjOkbX%2FIBYLvt4Wl2mgWWOanq3AtHcv2SfgMIYOG0hG5m%2BWHfgXBrzGN4cxeROZOguzAkzfXWvnXdjHVBU6VMmQRzjgAZbE%2F8SdcH565aETCc7NnKidak2zgixpkHI%2Fp6he1nCYfbHWeHGppmD8df%2FqawK4ER2HNy1vFjeV4UqTOq0ruIvuWcjRnZtjy9%2F0YJ3aRn8auJbAlkKvEHLeG5LzgHnECr0%2BHboKUP%2BwfVet3S%2BhWkCBg9llQPh31lSeU5bkM06Bib4mKo751lmfr4MZejIBoqV2napTZXSPvWLtpUYJoARyYTJCpfctUC6zY044%2BMwfQ%3D%3D: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /wandb-artifacts-prod/wandb_artifacts/129557231/688689262/a84ad5dd9242be0bd32175ceeb27ea74?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=Lo5cNnJP7Dk882JjOkbX%2FIBYLvt4Wl2mgWWOanq3AtHcv2SfgMIYOG0hG5m%2BWHfgXBrzGN4cxeROZOguzAkzfXWvnXdjHVBU6VMmQRzjgAZbE%2F8SdcH565aETCc7NnKidak2zgixpkHI%2Fp6he1nCYfbHWeHGppmD8df%2FqawK4ER2HNy1vFjeV4UqTOq0ruIvuWcjRnZtjy9%2F0YJ3aRn8auJbAlkKvEHLeG5LzgHnECr0%2BHboKUP%2BwfVet3S%2BhWkCBg9llQPh31lSeU5bkM06Bib4mKo751lmfr4MZejIBoqV2napTZXSPvWLtpUYJoARyYTJCpfctUC6zY044%2BMwfQ%3D%3D (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))
2024-01-12 03:06:30,684 ERROR Thread-77 :1657 [internal_api.py:upload_file():1696] upload_file request headers: {'User-Agent': 'python-requests/2.31.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Content-MD5': 'qErV3ZJCvgvTIXXO6yfqdA==', 'Content-Type': 'text/x-python; charset=utf-8', 'Content-Length': '7266'}
2024-01-12 03:06:30,684 ERROR Thread-77 :1657 [internal_api.py:upload_file():1698] upload_file response body:
2024-01-12 03:06:30,691 ERROR Thread-49 :1657 [internal_api.py:upload_file():1694] upload_file exception https://storage.googleapis.com/wandb-artifacts-prod/wandb_artifacts/129557231/688689262/bb1be7fd766b5e3ea2f9fe45f38631a4?Expires=1705115175&GoogleAccessId=gorilla-files-url-signer-man%40wandb-production.iam.gserviceaccount.com&Signature=filLj6XN%2F%2FoOHPzWeOA0R%2BMsrLCbYEJQrOnmoIIUwvfCclBAJAtgAO0LKHR8sjQ3MMoPBy822AAHSIdO1fDXBaF9PiH1eUWX145qKSOkxVShbCYwEQ3p3WiAjqG6DvDNnbo1v5VZXPR0i8W9ZY2TqLoWujp%2Bsl4iPdUFPh0yFHKHD%2BWSdLX0EBQ8xQJxrqP%2FsZuWNkd8ZPqfDX7FyTqZ8RTOSm%2Frf%2BgHdDLnjmSuosipDd7t10iiwdBNPoGsac%2Bj2KmelAa0e8meUf%2Bmjf1ZVtzeBJ6HIHgm6iBKNQqyq27cBficjdCE%2B4lm9kPX4q0e67DMs6Zp8UftIOicAeqpzg%3D%3D: