FileNotFoundError: [Errno 2] No such file or directory

l-scala · November 2, 2023, 12:22pm

FileNotFoundError: [Errno 2] No such file or directory: ‘C:\Users\lscala\AppData\Local\Temp\tmpack7yd_6\Samples / training samples.table.json’

Hello I am trying to run a wandb_session in Opensoundscape with wandb.init() within my model training to log training metrics but I get the above error and my code crashes without training.

import wandb
#Import WandB Utility
try:
    wandb.login()
    wandb_session = wandb.init(
        entity='seiche',
        project='multispecies_CNN',
        name='multispecies_CNN',
    )
except: #if wandb.init fails, don't use wandb logging
    print('failed to create wandb session. wandb session will be None')
    wandb_session = None

model.train(
    balanced_train_df,
    valid_df,
    epochs= 1,
    batch_size= 64,
    log_interval = 1,
    num_workers= 1, # if multiple processors are available utilise them to speed up training
    wandb_session = wandb_session,
    save_interval= 2,
    save_path= checkpoint_folder #location to save checkpoints
)

Please can you help resolve this error?

raphael-sanandres · November 3, 2023, 9:48pm

Hello @l-scala!

Could you send me your debug logs? They should be located in the wandb folder in your computer’s working directory. That folder has folders formatted as run-DATETIME-ID - each of which is associated with an individual run. Could you retrieve the debug.log and debug-internal.log files from one of these folders, specifically from the run that is erroring out?

l-scala · November 4, 2023, 10:07am

DEBUG LOG:

2023-11-04 10:05:11,543 INFO StreamThr :16808 [internal.py:wandb_internal():87] W&B internal server running at pid: 16808, started at: 2023-11-04 10:05:11.542620
2023-11-04 10:05:11,545 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: status
2023-11-04 10:05:11,559 INFO WriterThread:16808 [datastore.py:open_for_write():85] open: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\run-wkx1fbil.wandb
2023-11-04 10:05:11,561 DEBUG SenderThread:16808 [sender.py:send():336] send: header
2023-11-04 10:05:11,561 DEBUG SenderThread:16808 [sender.py:send():336] send: run
2023-11-04 10:05:12,720 INFO SenderThread:16808 [dir_watcher.py:init():219] watching files in: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files
2023-11-04 10:05:12,720 INFO SenderThread:16808 [sender.py:_start_run_threads():1078] run started: wkx1fbil with start time 1699092311.548622
2023-11-04 10:05:12,720 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: summary_record
2023-11-04 10:05:12,721 INFO SenderThread:16808 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
2023-11-04 10:05:12,721 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: check_version
2023-11-04 10:05:12,721 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: check_version
2023-11-04 10:05:13,251 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: run_start
2023-11-04 10:05:13,265 DEBUG HandlerThread:16808 [system_info.py:init():31] System info init
2023-11-04 10:05:13,265 DEBUG HandlerThread:16808 [system_info.py:init():46] System info init done
2023-11-04 10:05:13,265 INFO HandlerThread:16808 [system_monitor.py:start():183] Starting system monitor
2023-11-04 10:05:13,266 INFO SystemMonitor:16808 [system_monitor.py:_start():147] Starting system asset monitoring threads
2023-11-04 10:05:13,266 INFO HandlerThread:16808 [system_monitor.py:probe():204] Collecting system info
2023-11-04 10:05:13,273 INFO SystemMonitor:16808 [interfaces.py:start():187] Started cpu monitoring
2023-11-04 10:05:13,274 DEBUG HandlerThread:16808 [system_info.py:probe():195] Probing system
2023-11-04 10:05:13,274 INFO SystemMonitor:16808 [interfaces.py:start():187] Started disk monitoring
2023-11-04 10:05:13,275 INFO SystemMonitor:16808 [interfaces.py:start():187] Started memory monitoring
2023-11-04 10:05:13,287 INFO SystemMonitor:16808 [interfaces.py:start():187] Started network monitoring
2023-11-04 10:05:13,288 DEBUG HandlerThread:16808 [git.py:repo():40] git repository is invalid
2023-11-04 10:05:13,288 DEBUG HandlerThread:16808 [system_info.py:probe():240] Probing system done
2023-11-04 10:05:13,288 DEBUG HandlerThread:16808 [system_monitor.py:probe():213] {‘os’: ‘Windows-10-10.0.19045-SP0’, ‘python’: ‘3.11.4’, ‘heartbeatAt’: ‘2023-11-04T10:05:13.274374’, ‘startedAt’: ‘2023-11-04T10:05:11.522624’, ‘docker’: None, ‘cuda’: None, ‘args’: None, ‘state’: ‘running’, ‘program’: ‘’, ‘host’: ‘SM-Lorenzo-LT-2’, ‘username’: ‘lscala’, ‘executable’: ‘c:\Users\lscala\AppData\Local\anaconda3\envs\opensoundscape\python.exe’, ‘cpu_count’: 8, ‘cpu_count_logical’: 8, ‘cpu_freq’: {‘current’: 2000.0, ‘min’: 0.0, ‘max’: 2000.0}, ‘cpu_freq_per_core’: [{‘current’: 2000.0, ‘min’: 0.0, ‘max’: 2000.0}], ‘disk’: {‘total’: 475.69237899780273, ‘used’: 425.98743438720703}, ‘memory’: {‘total’: 15.372367858886719}}
2023-11-04 10:05:13,288 INFO HandlerThread:16808 [system_monitor.py:probe():214] Finished collecting system info
2023-11-04 10:05:13,288 INFO HandlerThread:16808 [system_monitor.py:probe():217] Publishing system info
2023-11-04 10:05:13,288 DEBUG HandlerThread:16808 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment
2023-11-04 10:05:13,289 DEBUG HandlerThread:16808 [system_info.py:_save_pip():67] Saving pip packages done
2023-11-04 10:05:13,289 DEBUG HandlerThread:16808 [system_info.py:_save_conda():74] Saving list of conda packages installed into the current environment
2023-11-04 10:05:13,329 ERROR HandlerThread:16808 [system_info.py:_save_conda():85] Error saving conda packages: [WinError 2] The system cannot find the file specified
Traceback (most recent call last):
File “c:\Users\lscala\AppData\Local\anaconda3\envs\opensoundscape\Lib\site-packages\wandb\sdk\internal\system\system_info.py”, line 81, in _save_conda
subprocess.call(
File “c:\Users\lscala\AppData\Local\anaconda3\envs\opensoundscape\Lib\subprocess.py”, line 389, in call
with Popen(*popenargs, **kwargs) as p:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “c:\Users\lscala\AppData\Local\anaconda3\envs\opensoundscape\Lib\subprocess.py”, line 1026, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File “c:\Users\lscala\AppData\Local\anaconda3\envs\opensoundscape\Lib\subprocess.py”, line 1538, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] The system cannot find the file specified
2023-11-04 10:05:13,340 DEBUG HandlerThread:16808 [system_info.py:_save_conda():86] Saving conda packages done
2023-11-04 10:05:13,341 INFO HandlerThread:16808 [system_monitor.py:probe():219] Finished publishing system info
2023-11-04 10:05:13,348 DEBUG SenderThread:16808 [sender.py:send():336] send: files
2023-11-04 10:05:13,349 INFO SenderThread:16808 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now
2023-11-04 10:05:13,367 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: stop_status
2023-11-04 10:05:13,368 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: stop_status
2023-11-04 10:05:13,371 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: pause
2023-11-04 10:05:13,371 INFO HandlerThread:16808 [handler.py:handle_request_pause():716] stopping system metrics thread or process
2023-11-04 10:05:13,371 INFO HandlerThread:16808 [system_monitor.py:finish():193] Stopping system monitor
2023-11-04 10:05:13,372 DEBUG SystemMonitor:16808 [system_monitor.py:_start():161] Starting system metrics aggregation loop
2023-11-04 10:05:13,373 DEBUG SystemMonitor:16808 [system_monitor.py:_start():168] Finished system metrics aggregation loop
2023-11-04 10:05:13,373 DEBUG SystemMonitor:16808 [system_monitor.py:_start():172] Publishing last batch of metrics
2023-11-04 10:05:13,373 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined cpu monitor
2023-11-04 10:05:13,374 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined disk monitor
2023-11-04 10:05:13,374 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined memory monitor
2023-11-04 10:05:13,374 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined network monitor
2023-11-04 10:05:13,723 INFO Thread-17 :16808 [dir_watcher.py:_on_file_created():278] file/dir created: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\requirements.txt
2023-11-04 10:05:13,723 INFO Thread-17 :16808 [dir_watcher.py:_on_file_created():278] file/dir created: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\conda-environment.yaml
2023-11-04 10:05:13,724 INFO Thread-17 :16808 [dir_watcher.py:_on_file_created():278] file/dir created: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\wandb-summary.json
2023-11-04 10:05:13,724 INFO Thread-17 :16808 [dir_watcher.py:_on_file_created():278] file/dir created: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\wandb-metadata.json
2023-11-04 10:05:14,090 DEBUG SenderThread:16808 [sender.py:send():336] send: telemetry
2023-11-04 10:05:14,091 DEBUG SenderThread:16808 [sender.py:send():336] send: stats
2023-11-04 10:05:14,821 INFO wandb-upload_0:16808 [upload_job.py:push():138] Uploaded file C:\Users\lscala\AppData\Local\Temp\tmpza3rp9m8wandb\lwtk4y4t-wandb-metadata.json
2023-11-04 10:05:17,111 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: status_report
2023-11-04 10:05:17,824 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: resume
2023-11-04 10:05:17,824 INFO HandlerThread:16808 [handler.py:handle_request_resume():707] starting system metrics thread or process
2023-11-04 10:05:17,824 INFO HandlerThread:16808 [system_monitor.py:start():183] Starting system monitor
2023-11-04 10:05:17,825 INFO SystemMonitor:16808 [system_monitor.py:_start():147] Starting system asset monitoring threads
2023-11-04 10:05:17,827 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: pause
2023-11-04 10:05:17,827 INFO HandlerThread:16808 [handler.py:handle_request_pause():716] stopping system metrics thread or process
2023-11-04 10:05:17,827 INFO HandlerThread:16808 [system_monitor.py:finish():193] Stopping system monitor
2023-11-04 10:05:17,827 WARNING HandlerThread:16808 [interfaces.py:finish():204] Failed to finish cpu monitoring: cannot join thread before it is started
2023-11-04 10:05:17,827 INFO SystemMonitor:16808 [interfaces.py:start():187] Started cpu monitoring
2023-11-04 10:05:17,827 DEBUG SystemMonitor:16808 [system_monitor.py:_start():161] Starting system metrics aggregation loop
2023-11-04 10:05:17,827 DEBUG SystemMonitor:16808 [system_monitor.py:_start():168] Finished system metrics aggregation loop
2023-11-04 10:05:17,827 DEBUG SystemMonitor:16808 [system_monitor.py:_start():172] Publishing last batch of metrics
2023-11-04 10:05:22,166 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: status_report
2023-11-04 10:05:23,652 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: resume
2023-11-04 10:05:23,652 INFO HandlerThread:16808 [handler.py:handle_request_resume():707] starting system metrics thread or process
2023-11-04 10:05:23,652 INFO HandlerThread:16808 [system_monitor.py:start():183] Starting system monitor
2023-11-04 10:05:23,653 INFO SystemMonitor:16808 [system_monitor.py:_start():147] Starting system asset monitoring threads
2023-11-04 10:05:23,660 INFO SystemMonitor:16808 [interfaces.py:start():187] Started cpu monitoring
2023-11-04 10:05:23,661 INFO SystemMonitor:16808 [interfaces.py:start():187] Started disk monitoring
2023-11-04 10:05:23,661 DEBUG SenderThread:16808 [sender.py:send():336] send: config
2023-11-04 10:05:23,662 DEBUG SenderThread:16808 [sender.py:send():336] send: config
2023-11-04 10:05:23,663 INFO SystemMonitor:16808 [interfaces.py:start():187] Started memory monitoring
2023-11-04 10:05:23,663 DEBUG SenderThread:16808 [sender.py:send():336] send: telemetry
2023-11-04 10:05:23,663 INFO SystemMonitor:16808 [interfaces.py:start():187] Started network monitoring
2023-11-04 10:05:23,827 INFO Thread-17 :16808 [dir_watcher.py:_on_file_created():278] file/dir created: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\output.log
2023-11-04 10:05:25,843 INFO Thread-17 :16808 [dir_watcher.py:_on_file_modified():295] file/dir modified: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\output.log
2023-11-04 10:05:27,295 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: pause
2023-11-04 10:05:27,295 INFO HandlerThread:16808 [handler.py:handle_request_pause():716] stopping system metrics thread or process
2023-11-04 10:05:27,295 INFO HandlerThread:16808 [system_monitor.py:finish():193] Stopping system monitor
2023-11-04 10:05:27,296 DEBUG SystemMonitor:16808 [system_monitor.py:_start():161] Starting system metrics aggregation loop
2023-11-04 10:05:27,296 DEBUG SystemMonitor:16808 [system_monitor.py:_start():168] Finished system metrics aggregation loop
2023-11-04 10:05:27,296 DEBUG SystemMonitor:16808 [system_monitor.py:_start():172] Publishing last batch of metrics
2023-11-04 10:05:27,296 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined cpu monitor
2023-11-04 10:05:27,296 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined disk monitor
2023-11-04 10:05:27,296 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined memory monitor
2023-11-04 10:05:27,296 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined network monitor
2023-11-04 10:05:27,297 DEBUG SenderThread:16808 [sender.py:send():336] send: stats
2023-11-04 10:05:27,297 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: status_report
2023-11-04 10:05:27,864 INFO Thread-17 :16808 [dir_watcher.py:_on_file_modified():295] file/dir modified: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\output.log
2023-11-04 10:05:28,361 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: keepalive
2023-11-04 10:05:28,362 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: keepalive
2023-11-04 10:05:32,358 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: status_report
2023-11-04 10:05:33,406 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: keepalive
2023-11-04 10:05:35,799 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: resume
2023-11-04 10:05:35,799 INFO HandlerThread:16808 [handler.py:handle_request_resume():707] starting system metrics thread or process
2023-11-04 10:05:35,799 INFO HandlerThread:16808 [system_monitor.py:start():183] Starting system monitor
2023-11-04 10:05:35,800 INFO SystemMonitor:16808 [system_monitor.py:_start():147] Starting system asset monitoring threads
2023-11-04 10:05:35,801 DEBUG SenderThread:16808 [sender.py:send():336] send: telemetry
2023-11-04 10:05:35,808 INFO SystemMonitor:16808 [interfaces.py:start():187] Started cpu monitoring
2023-11-04 10:05:35,809 DEBUG SenderThread:16808 [sender.py:send():336] send: exit
2023-11-04 10:05:35,809 INFO SystemMonitor:16808 [interfaces.py:start():187] Started disk monitoring
2023-11-04 10:05:35,809 INFO SenderThread:16808 [sender.py:send_exit():559] handling exit code: 0
2023-11-04 10:05:35,810 INFO SenderThread:16808 [sender.py:send_exit():561] handling runtime: 3
2023-11-04 10:05:35,810 INFO SystemMonitor:16808 [interfaces.py:start():187] Started memory monitoring
2023-11-04 10:05:35,825 INFO SystemMonitor:16808 [interfaces.py:start():187] Started network monitoring
2023-11-04 10:05:35,826 INFO SenderThread:16808 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
2023-11-04 10:05:35,826 INFO SenderThread:16808 [sender.py:send_exit():567] send defer
2023-11-04 10:05:35,827 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:35,827 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 0
2023-11-04 10:05:35,827 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:35,827 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 0
2023-11-04 10:05:35,827 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 1
2023-11-04 10:05:35,827 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:35,827 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 1
2023-11-04 10:05:35,828 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:35,828 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 1
2023-11-04 10:05:35,828 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 2
2023-11-04 10:05:35,828 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:35,828 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 2
2023-11-04 10:05:35,828 INFO HandlerThread:16808 [system_monitor.py:finish():193] Stopping system monitor
2023-11-04 10:05:35,828 DEBUG SystemMonitor:16808 [system_monitor.py:_start():161] Starting system metrics aggregation loop
2023-11-04 10:05:35,829 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined cpu monitor
2023-11-04 10:05:35,829 DEBUG SystemMonitor:16808 [system_monitor.py:_start():168] Finished system metrics aggregation loop
2023-11-04 10:05:35,829 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined disk monitor
2023-11-04 10:05:35,829 DEBUG SystemMonitor:16808 [system_monitor.py:_start():172] Publishing last batch of metrics
2023-11-04 10:05:35,830 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined memory monitor
2023-11-04 10:05:35,830 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined network monitor
2023-11-04 10:05:35,831 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:35,831 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 2
2023-11-04 10:05:35,831 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 3
2023-11-04 10:05:35,831 DEBUG SenderThread:16808 [sender.py:send():336] send: stats
2023-11-04 10:05:35,831 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:35,831 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 3
2023-11-04 10:05:35,831 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:35,832 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 3
2023-11-04 10:05:35,832 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 4
2023-11-04 10:05:35,832 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:35,832 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 4
2023-11-04 10:05:35,832 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:35,832 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 4
2023-11-04 10:05:35,832 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 5
2023-11-04 10:05:35,833 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:35,833 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 5
2023-11-04 10:05:35,833 DEBUG SenderThread:16808 [sender.py:send():336] send: summary
2023-11-04 10:05:35,834 INFO SenderThread:16808 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
2023-11-04 10:05:35,834 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:35,834 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 5
2023-11-04 10:05:35,834 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 6
2023-11-04 10:05:35,834 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:35,834 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 6
2023-11-04 10:05:35,835 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:35,835 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 6
2023-11-04 10:05:35,838 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: status_report
2023-11-04 10:05:35,937 INFO Thread-17 :16808 [dir_watcher.py:_on_file_modified():295] file/dir modified: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\wandb-summary.json
2023-11-04 10:05:36,410 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 7
2023-11-04 10:05:36,411 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:36,411 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 7
2023-11-04 10:05:36,411 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:36,411 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 7
2023-11-04 10:05:36,820 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: poll_exit
2023-11-04 10:05:36,944 INFO Thread-17 :16808 [dir_watcher.py:_on_file_modified():295] file/dir modified: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\config.yaml
2023-11-04 10:05:37,748 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 8
2023-11-04 10:05:37,749 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: poll_exit
2023-11-04 10:05:37,749 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:37,749 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 8
2023-11-04 10:05:37,750 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:37,750 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 8
2023-11-04 10:05:37,751 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 9
2023-11-04 10:05:37,751 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:37,751 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 9
2023-11-04 10:05:37,752 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:37,752 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 9
2023-11-04 10:05:37,752 INFO SenderThread:16808 [dir_watcher.py:finish():365] shutting down directory watcher
2023-11-04 10:05:37,833 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: poll_exit
2023-11-04 10:05:37,959 INFO SenderThread:16808 [dir_watcher.py:_on_file_modified():295] file/dir modified: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\output.log
2023-11-04 10:05:37,960 INFO SenderThread:16808 [dir_watcher.py:finish():395] scan: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files
2023-11-04 10:05:37,960 INFO SenderThread:16808 [dir_watcher.py:finish():409] scan save: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\conda-environment.yaml conda-environment.yaml
2023-11-04 10:05:37,961 INFO SenderThread:16808 [dir_watcher.py:finish():409] scan save: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\config.yaml config.yaml
2023-11-04 10:05:37,961 INFO SenderThread:16808 [dir_watcher.py:finish():409] scan save: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\output.log output.log
2023-11-04 10:05:37,964 INFO SenderThread:16808 [dir_watcher.py:finish():409] scan save: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\requirements.txt requirements.txt
2023-11-04 10:05:37,967 INFO SenderThread:16808 [dir_watcher.py:finish():409] scan save: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\wandb-metadata.json wandb-metadata.json
2023-11-04 10:05:37,967 INFO SenderThread:16808 [dir_watcher.py:finish():409] scan save: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\wandb-summary.json wandb-summary.json
2023-11-04 10:05:37,968 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 10
2023-11-04 10:05:37,968 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: poll_exit
2023-11-04 10:05:37,968 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:37,972 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 10
2023-11-04 10:05:37,974 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:37,975 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 10
2023-11-04 10:05:37,975 INFO SenderThread:16808 [file_pusher.py:finish():164] shutting down file pusher
2023-11-04 10:05:38,847 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: poll_exit
2023-11-04 10:05:38,847 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: poll_exit
2023-11-04 10:05:39,114 INFO wandb-upload_2:16808 [upload_job.py:push():138] Uploaded file c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\requirements.txt
2023-11-04 10:05:39,637 INFO wandb-upload_1:16808 [upload_job.py:push():138] Uploaded file c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\output.log
2023-11-04 10:05:39,657 INFO wandb-upload_3:16808 [upload_job.py:push():138] Uploaded file c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\wandb-summary.json
2023-11-04 10:05:39,752 INFO wandb-upload_0:16808 [upload_job.py:push():138] Uploaded file c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\config.yaml
2023-11-04 10:05:39,852 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: poll_exit
2023-11-04 10:05:39,852 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: poll_exit
2023-11-04 10:05:39,961 INFO Thread-16 (_thread_body):16808 [sender.py:transition_state():587] send defer: 11
2023-11-04 10:05:39,961 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:39,961 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 11
2023-11-04 10:05:39,962 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:39,962 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 11
2023-11-04 10:05:39,962 INFO SenderThread:16808 [file_pusher.py:join():169] waiting for file pusher
2023-11-04 10:05:39,962 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 12
2023-11-04 10:05:39,962 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:39,962 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 12
2023-11-04 10:05:39,962 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:39,963 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 12
2023-11-04 10:05:40,123 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 13
2023-11-04 10:05:40,123 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:40,123 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 13
2023-11-04 10:05:40,124 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:40,124 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 13
2023-11-04 10:05:40,124 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 14
2023-11-04 10:05:40,124 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:40,124 DEBUG SenderThread:16808 [sender.py:send():336] send: final
2023-11-04 10:05:40,124 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 14
2023-11-04 10:05:40,125 DEBUG SenderThread:16808 [sender.py:send():336] send: footer
2023-11-04 10:05:40,125 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:40,125 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 14
2023-11-04 10:05:40,126 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: poll_exit
2023-11-04 10:05:40,126 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: server_info
2023-11-04 10:05:40,126 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: get_summary
2023-11-04 10:05:40,127 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: sampled_history
2023-11-04 10:05:40,127 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: poll_exit
2023-11-04 10:05:40,127 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: server_info
2023-11-04 10:05:41,188 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: shutdown
2023-11-04 10:05:41,188 INFO HandlerThread:16808 [handler.py:finish():842] shutting down handler
2023-11-04 10:05:42,148 INFO WriterThread:16808 [datastore.py:close():298] close: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\run-wkx1fbil.wandb
2023-11-04 10:05:42,196 INFO SenderThread:16808 [sender.py:finish():1504] shutting down sender
2023-11-04 10:05:42,196 INFO SenderThread:16808 [file_pusher.py:finish():164] shutting down file pusher
2023-11-04 10:05:42,196 INFO SenderThread:16808 [file_pusher.py:join():169] waiting for file pusher

l-scala · November 4, 2023, 10:13am

DEBUG LOG INTERNAL

2023-11-04 10:05:11,543 INFO StreamThr :16808 [internal.py:wandb_internal():87] W&B internal server running at pid: 16808, started at: 2023-11-04 10:05:11.542620
2023-11-04 10:05:11,545 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: status
2023-11-04 10:05:11,559 INFO WriterThread:16808 [datastore.py:open_for_write():85] open: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\run-wkx1fbil.wandb
2023-11-04 10:05:11,561 DEBUG SenderThread:16808 [sender.py:send():336] send: header
2023-11-04 10:05:11,561 DEBUG SenderThread:16808 [sender.py:send():336] send: run
2023-11-04 10:05:12,720 INFO SenderThread:16808 [dir_watcher.py:init():219] watching files in: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files
2023-11-04 10:05:12,720 INFO SenderThread:16808 [sender.py:_start_run_threads():1078] run started: wkx1fbil with start time 1699092311.548622
2023-11-04 10:05:12,720 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: summary_record
2023-11-04 10:05:12,721 INFO SenderThread:16808 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
2023-11-04 10:05:12,721 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: check_version
2023-11-04 10:05:12,721 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: check_version
2023-11-04 10:05:13,251 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: run_start
2023-11-04 10:05:13,265 DEBUG HandlerThread:16808 [system_info.py:init():31] System info init
2023-11-04 10:05:13,265 DEBUG HandlerThread:16808 [system_info.py:init():46] System info init done
2023-11-04 10:05:13,265 INFO HandlerThread:16808 [system_monitor.py:start():183] Starting system monitor
2023-11-04 10:05:13,266 INFO SystemMonitor:16808 [system_monitor.py:_start():147] Starting system asset monitoring threads
2023-11-04 10:05:13,266 INFO HandlerThread:16808 [system_monitor.py:probe():204] Collecting system info
2023-11-04 10:05:13,273 INFO SystemMonitor:16808 [interfaces.py:start():187] Started cpu monitoring
2023-11-04 10:05:13,274 DEBUG HandlerThread:16808 [system_info.py:probe():195] Probing system
2023-11-04 10:05:13,274 INFO SystemMonitor:16808 [interfaces.py:start():187] Started disk monitoring
2023-11-04 10:05:13,275 INFO SystemMonitor:16808 [interfaces.py:start():187] Started memory monitoring
2023-11-04 10:05:13,287 INFO SystemMonitor:16808 [interfaces.py:start():187] Started network monitoring
2023-11-04 10:05:13,288 DEBUG HandlerThread:16808 [git.py:repo():40] git repository is invalid
2023-11-04 10:05:13,288 DEBUG HandlerThread:16808 [system_info.py:probe():240] Probing system done
2023-11-04 10:05:13,288 DEBUG HandlerThread:16808 [system_monitor.py:probe():213] {‘os’: ‘Windows-10-10.0.19045-SP0’, ‘python’: ‘3.11.4’, ‘heartbeatAt’: ‘2023-11-04T10:05:13.274374’, ‘startedAt’: ‘2023-11-04T10:05:11.522624’, ‘docker’: None, ‘cuda’: None, ‘args’: None, ‘state’: ‘running’, ‘program’: ‘’, ‘host’: ‘SM-Lorenzo-LT-2’, ‘username’: ‘lscala’, ‘executable’: ‘c:\Users\lscala\AppData\Local\anaconda3\envs\opensoundscape\python.exe’, ‘cpu_count’: 8, ‘cpu_count_logical’: 8, ‘cpu_freq’: {‘current’: 2000.0, ‘min’: 0.0, ‘max’: 2000.0}, ‘cpu_freq_per_core’: [{‘current’: 2000.0, ‘min’: 0.0, ‘max’: 2000.0}], ‘disk’: {‘total’: 475.69237899780273, ‘used’: 425.98743438720703}, ‘memory’: {‘total’: 15.372367858886719}}
2023-11-04 10:05:13,288 INFO HandlerThread:16808 [system_monitor.py:probe():214] Finished collecting system info
2023-11-04 10:05:13,288 INFO HandlerThread:16808 [system_monitor.py:probe():217] Publishing system info
2023-11-04 10:05:13,288 DEBUG HandlerThread:16808 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment
2023-11-04 10:05:13,289 DEBUG HandlerThread:16808 [system_info.py:_save_pip():67] Saving pip packages done
2023-11-04 10:05:13,289 DEBUG HandlerThread:16808 [system_info.py:_save_conda():74] Saving list of conda packages installed into the current environment
2023-11-04 10:05:13,329 ERROR HandlerThread:16808 [system_info.py:_save_conda():85] Error saving conda packages: [WinError 2] The system cannot find the file specified
Traceback (most recent call last):
File “c:\Users\lscala\AppData\Local\anaconda3\envs\opensoundscape\Lib\site-packages\wandb\sdk\internal\system\system_info.py”, line 81, in _save_conda
subprocess.call(
File “c:\Users\lscala\AppData\Local\anaconda3\envs\opensoundscape\Lib\subprocess.py”, line 389, in call
with Popen(*popenargs, **kwargs) as p:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “c:\Users\lscala\AppData\Local\anaconda3\envs\opensoundscape\Lib\subprocess.py”, line 1026, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File “c:\Users\lscala\AppData\Local\anaconda3\envs\opensoundscape\Lib\subprocess.py”, line 1538, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] The system cannot find the file specified
2023-11-04 10:05:13,340 DEBUG HandlerThread:16808 [system_info.py:_save_conda():86] Saving conda packages done
2023-11-04 10:05:13,341 INFO HandlerThread:16808 [system_monitor.py:probe():219] Finished publishing system info
2023-11-04 10:05:13,348 DEBUG SenderThread:16808 [sender.py:send():336] send: files
2023-11-04 10:05:13,349 INFO SenderThread:16808 [sender.py:_save_file():1332] saving file wandb-metadata.json with policy now
2023-11-04 10:05:13,367 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: stop_status
2023-11-04 10:05:13,368 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: stop_status
2023-11-04 10:05:13,371 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: pause
2023-11-04 10:05:13,371 INFO HandlerThread:16808 [handler.py:handle_request_pause():716] stopping system metrics thread or process
2023-11-04 10:05:13,371 INFO HandlerThread:16808 [system_monitor.py:finish():193] Stopping system monitor
2023-11-04 10:05:13,372 DEBUG SystemMonitor:16808 [system_monitor.py:_start():161] Starting system metrics aggregation loop
2023-11-04 10:05:13,373 DEBUG SystemMonitor:16808 [system_monitor.py:_start():168] Finished system metrics aggregation loop
2023-11-04 10:05:13,373 DEBUG SystemMonitor:16808 [system_monitor.py:_start():172] Publishing last batch of metrics
2023-11-04 10:05:13,373 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined cpu monitor
2023-11-04 10:05:13,374 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined disk monitor
2023-11-04 10:05:13,374 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined memory monitor
2023-11-04 10:05:13,374 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined network monitor
2023-11-04 10:05:13,723 INFO Thread-17 :16808 [dir_watcher.py:_on_file_created():278] file/dir created: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\requirements.txt
2023-11-04 10:05:13,723 INFO Thread-17 :16808 [dir_watcher.py:_on_file_created():278] file/dir created: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\conda-environment.yaml
2023-11-04 10:05:13,724 INFO Thread-17 :16808 [dir_watcher.py:_on_file_created():278] file/dir created: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\wandb-summary.json
2023-11-04 10:05:13,724 INFO Thread-17 :16808 [dir_watcher.py:_on_file_created():278] file/dir created: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\wandb-metadata.json
2023-11-04 10:05:14,090 DEBUG SenderThread:16808 [sender.py:send():336] send: telemetry
2023-11-04 10:05:14,091 DEBUG SenderThread:16808 [sender.py:send():336] send: stats
2023-11-04 10:05:14,821 INFO wandb-upload_0:16808 [upload_job.py:push():138] Uploaded file C:\Users\lscala\AppData\Local\Temp\tmpza3rp9m8wandb\lwtk4y4t-wandb-metadata.json
2023-11-04 10:05:17,111 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: status_report
2023-11-04 10:05:17,824 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: resume
2023-11-04 10:05:17,824 INFO HandlerThread:16808 [handler.py:handle_request_resume():707] starting system metrics thread or process
2023-11-04 10:05:17,824 INFO HandlerThread:16808 [system_monitor.py:start():183] Starting system monitor
2023-11-04 10:05:17,825 INFO SystemMonitor:16808 [system_monitor.py:_start():147] Starting system asset monitoring threads
2023-11-04 10:05:17,827 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: pause
2023-11-04 10:05:17,827 INFO HandlerThread:16808 [handler.py:handle_request_pause():716] stopping system metrics thread or process
2023-11-04 10:05:17,827 INFO HandlerThread:16808 [system_monitor.py:finish():193] Stopping system monitor
2023-11-04 10:05:17,827 WARNING HandlerThread:16808 [interfaces.py:finish():204] Failed to finish cpu monitoring: cannot join thread before it is started
2023-11-04 10:05:17,827 INFO SystemMonitor:16808 [interfaces.py:start():187] Started cpu monitoring
2023-11-04 10:05:17,827 DEBUG SystemMonitor:16808 [system_monitor.py:_start():161] Starting system metrics aggregation loop
2023-11-04 10:05:17,827 DEBUG SystemMonitor:16808 [system_monitor.py:_start():168] Finished system metrics aggregation loop
2023-11-04 10:05:17,827 DEBUG SystemMonitor:16808 [system_monitor.py:_start():172] Publishing last batch of metrics
2023-11-04 10:05:22,166 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: status_report
2023-11-04 10:05:23,652 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: resume
2023-11-04 10:05:23,652 INFO HandlerThread:16808 [handler.py:handle_request_resume():707] starting system metrics thread or process
2023-11-04 10:05:23,652 INFO HandlerThread:16808 [system_monitor.py:start():183] Starting system monitor
2023-11-04 10:05:23,653 INFO SystemMonitor:16808 [system_monitor.py:_start():147] Starting system asset monitoring threads
2023-11-04 10:05:23,660 INFO SystemMonitor:16808 [interfaces.py:start():187] Started cpu monitoring
2023-11-04 10:05:23,661 INFO SystemMonitor:16808 [interfaces.py:start():187] Started disk monitoring
2023-11-04 10:05:23,661 DEBUG SenderThread:16808 [sender.py:send():336] send: config
2023-11-04 10:05:23,662 DEBUG SenderThread:16808 [sender.py:send():336] send: config
2023-11-04 10:05:23,663 INFO SystemMonitor:16808 [interfaces.py:start():187] Started memory monitoring
2023-11-04 10:05:23,663 DEBUG SenderThread:16808 [sender.py:send():336] send: telemetry
2023-11-04 10:05:23,663 INFO SystemMonitor:16808 [interfaces.py:start():187] Started network monitoring
2023-11-04 10:05:23,827 INFO Thread-17 :16808 [dir_watcher.py:_on_file_created():278] file/dir created: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\output.log
2023-11-04 10:05:25,843 INFO Thread-17 :16808 [dir_watcher.py:_on_file_modified():295] file/dir modified: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\output.log
2023-11-04 10:05:27,295 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: pause
2023-11-04 10:05:27,295 INFO HandlerThread:16808 [handler.py:handle_request_pause():716] stopping system metrics thread or process
2023-11-04 10:05:27,295 INFO HandlerThread:16808 [system_monitor.py:finish():193] Stopping system monitor
2023-11-04 10:05:27,296 DEBUG SystemMonitor:16808 [system_monitor.py:_start():161] Starting system metrics aggregation loop
2023-11-04 10:05:27,296 DEBUG SystemMonitor:16808 [system_monitor.py:_start():168] Finished system metrics aggregation loop
2023-11-04 10:05:27,296 DEBUG SystemMonitor:16808 [system_monitor.py:_start():172] Publishing last batch of metrics
2023-11-04 10:05:27,296 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined cpu monitor
2023-11-04 10:05:27,296 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined disk monitor
2023-11-04 10:05:27,296 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined memory monitor
2023-11-04 10:05:27,296 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined network monitor
2023-11-04 10:05:27,297 DEBUG SenderThread:16808 [sender.py:send():336] send: stats
2023-11-04 10:05:27,297 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: status_report
2023-11-04 10:05:27,864 INFO Thread-17 :16808 [dir_watcher.py:_on_file_modified():295] file/dir modified: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\output.log
2023-11-04 10:05:28,361 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: keepalive
2023-11-04 10:05:28,362 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: keepalive
2023-11-04 10:05:32,358 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: status_report
2023-11-04 10:05:33,406 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: keepalive
2023-11-04 10:05:35,799 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: resume
2023-11-04 10:05:35,799 INFO HandlerThread:16808 [handler.py:handle_request_resume():707] starting system metrics thread or process
2023-11-04 10:05:35,799 INFO HandlerThread:16808 [system_monitor.py:start():183] Starting system monitor
2023-11-04 10:05:35,800 INFO SystemMonitor:16808 [system_monitor.py:_start():147] Starting system asset monitoring threads
2023-11-04 10:05:35,801 DEBUG SenderThread:16808 [sender.py:send():336] send: telemetry
2023-11-04 10:05:35,808 INFO SystemMonitor:16808 [interfaces.py:start():187] Started cpu monitoring
2023-11-04 10:05:35,809 DEBUG SenderThread:16808 [sender.py:send():336] send: exit
2023-11-04 10:05:35,809 INFO SystemMonitor:16808 [interfaces.py:start():187] Started disk monitoring
2023-11-04 10:05:35,809 INFO SenderThread:16808 [sender.py:send_exit():559] handling exit code: 0
2023-11-04 10:05:35,810 INFO SenderThread:16808 [sender.py:send_exit():561] handling runtime: 3
2023-11-04 10:05:35,810 INFO SystemMonitor:16808 [interfaces.py:start():187] Started memory monitoring
2023-11-04 10:05:35,825 INFO SystemMonitor:16808 [interfaces.py:start():187] Started network monitoring
2023-11-04 10:05:35,826 INFO SenderThread:16808 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
2023-11-04 10:05:35,826 INFO SenderThread:16808 [sender.py:send_exit():567] send defer
2023-11-04 10:05:35,827 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:35,827 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 0
2023-11-04 10:05:35,827 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:35,827 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 0
2023-11-04 10:05:35,827 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 1
2023-11-04 10:05:35,827 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:35,827 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 1
2023-11-04 10:05:35,828 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:35,828 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 1
2023-11-04 10:05:35,828 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 2
2023-11-04 10:05:35,828 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:35,828 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 2
2023-11-04 10:05:35,828 INFO HandlerThread:16808 [system_monitor.py:finish():193] Stopping system monitor
2023-11-04 10:05:35,828 DEBUG SystemMonitor:16808 [system_monitor.py:_start():161] Starting system metrics aggregation loop
2023-11-04 10:05:35,829 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined cpu monitor
2023-11-04 10:05:35,829 DEBUG SystemMonitor:16808 [system_monitor.py:_start():168] Finished system metrics aggregation loop
2023-11-04 10:05:35,829 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined disk monitor
2023-11-04 10:05:35,829 DEBUG SystemMonitor:16808 [system_monitor.py:_start():172] Publishing last batch of metrics
2023-11-04 10:05:35,830 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined memory monitor
2023-11-04 10:05:35,830 INFO HandlerThread:16808 [interfaces.py:finish():199] Joined network monitor
2023-11-04 10:05:35,831 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:35,831 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 2
2023-11-04 10:05:35,831 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 3
2023-11-04 10:05:35,831 DEBUG SenderThread:16808 [sender.py:send():336] send: stats
2023-11-04 10:05:35,831 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:35,831 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 3
2023-11-04 10:05:35,831 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:35,832 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 3
2023-11-04 10:05:35,832 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 4
2023-11-04 10:05:35,832 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:35,832 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 4
2023-11-04 10:05:35,832 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:35,832 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 4
2023-11-04 10:05:35,832 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 5
2023-11-04 10:05:35,833 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:35,833 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 5
2023-11-04 10:05:35,833 DEBUG SenderThread:16808 [sender.py:send():336] send: summary
2023-11-04 10:05:35,834 INFO SenderThread:16808 [sender.py:_save_file():1332] saving file wandb-summary.json with policy end
2023-11-04 10:05:35,834 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:35,834 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 5
2023-11-04 10:05:35,834 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 6
2023-11-04 10:05:35,834 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:35,834 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 6
2023-11-04 10:05:35,835 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:35,835 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 6
2023-11-04 10:05:35,838 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: status_report
2023-11-04 10:05:35,937 INFO Thread-17 :16808 [dir_watcher.py:_on_file_modified():295] file/dir modified: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\wandb-summary.json
2023-11-04 10:05:36,410 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 7
2023-11-04 10:05:36,411 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:36,411 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 7
2023-11-04 10:05:36,411 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:36,411 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 7
2023-11-04 10:05:36,820 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: poll_exit
2023-11-04 10:05:36,944 INFO Thread-17 :16808 [dir_watcher.py:_on_file_modified():295] file/dir modified: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\config.yaml
2023-11-04 10:05:37,748 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 8
2023-11-04 10:05:37,749 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: poll_exit
2023-11-04 10:05:37,749 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:37,749 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 8
2023-11-04 10:05:37,750 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:37,750 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 8
2023-11-04 10:05:37,751 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 9
2023-11-04 10:05:37,751 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:37,751 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 9
2023-11-04 10:05:37,752 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:37,752 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 9
2023-11-04 10:05:37,752 INFO SenderThread:16808 [dir_watcher.py:finish():365] shutting down directory watcher
2023-11-04 10:05:37,833 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: poll_exit
2023-11-04 10:05:37,959 INFO SenderThread:16808 [dir_watcher.py:_on_file_modified():295] file/dir modified: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\output.log
2023-11-04 10:05:37,960 INFO SenderThread:16808 [dir_watcher.py:finish():395] scan: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files
2023-11-04 10:05:37,960 INFO SenderThread:16808 [dir_watcher.py:finish():409] scan save: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\conda-environment.yaml conda-environment.yaml
2023-11-04 10:05:37,961 INFO SenderThread:16808 [dir_watcher.py:finish():409] scan save: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\config.yaml config.yaml
2023-11-04 10:05:37,961 INFO SenderThread:16808 [dir_watcher.py:finish():409] scan save: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\output.log output.log
2023-11-04 10:05:37,964 INFO SenderThread:16808 [dir_watcher.py:finish():409] scan save: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\requirements.txt requirements.txt
2023-11-04 10:05:37,967 INFO SenderThread:16808 [dir_watcher.py:finish():409] scan save: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\wandb-metadata.json wandb-metadata.json
2023-11-04 10:05:37,967 INFO SenderThread:16808 [dir_watcher.py:finish():409] scan save: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\wandb-summary.json wandb-summary.json
2023-11-04 10:05:37,968 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 10
2023-11-04 10:05:37,968 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: poll_exit
2023-11-04 10:05:37,968 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:37,972 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 10
2023-11-04 10:05:37,974 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:37,975 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 10
2023-11-04 10:05:37,975 INFO SenderThread:16808 [file_pusher.py:finish():164] shutting down file pusher
2023-11-04 10:05:38,847 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: poll_exit
2023-11-04 10:05:38,847 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: poll_exit
2023-11-04 10:05:39,114 INFO wandb-upload_2:16808 [upload_job.py:push():138] Uploaded file c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\requirements.txt
2023-11-04 10:05:39,637 INFO wandb-upload_1:16808 [upload_job.py:push():138] Uploaded file c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\output.log
2023-11-04 10:05:39,657 INFO wandb-upload_3:16808 [upload_job.py:push():138] Uploaded file c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\wandb-summary.json
2023-11-04 10:05:39,752 INFO wandb-upload_0:16808 [upload_job.py:push():138] Uploaded file c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\files\config.yaml
2023-11-04 10:05:39,852 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: poll_exit
2023-11-04 10:05:39,852 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: poll_exit
2023-11-04 10:05:39,961 INFO Thread-16 (_thread_body):16808 [sender.py:transition_state():587] send defer: 11
2023-11-04 10:05:39,961 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:39,961 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 11
2023-11-04 10:05:39,962 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:39,962 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 11
2023-11-04 10:05:39,962 INFO SenderThread:16808 [file_pusher.py:join():169] waiting for file pusher
2023-11-04 10:05:39,962 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 12
2023-11-04 10:05:39,962 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:39,962 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 12
2023-11-04 10:05:39,962 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:39,963 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 12
2023-11-04 10:05:40,123 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 13
2023-11-04 10:05:40,123 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:40,123 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 13
2023-11-04 10:05:40,124 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:40,124 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 13
2023-11-04 10:05:40,124 INFO SenderThread:16808 [sender.py:transition_state():587] send defer: 14
2023-11-04 10:05:40,124 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: defer
2023-11-04 10:05:40,124 DEBUG SenderThread:16808 [sender.py:send():336] send: final
2023-11-04 10:05:40,124 INFO HandlerThread:16808 [handler.py:handle_request_defer():170] handle defer: 14
2023-11-04 10:05:40,125 DEBUG SenderThread:16808 [sender.py:send():336] send: footer
2023-11-04 10:05:40,125 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: defer
2023-11-04 10:05:40,125 INFO SenderThread:16808 [sender.py:send_request_defer():583] handle sender defer: 14
2023-11-04 10:05:40,126 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: poll_exit
2023-11-04 10:05:40,126 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: server_info
2023-11-04 10:05:40,126 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: get_summary
2023-11-04 10:05:40,127 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: sampled_history
2023-11-04 10:05:40,127 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: poll_exit
2023-11-04 10:05:40,127 DEBUG SenderThread:16808 [sender.py:send_request():363] send_request: server_info
2023-11-04 10:05:41,188 DEBUG HandlerThread:16808 [handler.py:handle_request():144] handle_request: shutdown
2023-11-04 10:05:41,188 INFO HandlerThread:16808 [handler.py:finish():842] shutting down handler
2023-11-04 10:05:42,148 INFO WriterThread:16808 [datastore.py:close():298] close: c:\Users\lscala\Python_learning\wandb\run-20231104_100511-wkx1fbil\run-wkx1fbil.wandb
2023-11-04 10:05:42,196 INFO SenderThread:16808 [sender.py:finish():1504] shutting down sender
2023-11-04 10:05:42,196 INFO SenderThread:16808 [file_pusher.py:finish():164] shutting down file pusher
2023-11-04 10:05:42,196 INFO SenderThread:16808 [file_pusher.py:join():169] waiting for file pusher

l-scala · November 8, 2023, 10:38am

hello- are the logs useful? thank you

raphael-sanandres · November 8, 2023, 10:58pm

Hello! After looking into the debug logs, it looks like this is rooted in a subprocess issue on opensournscape’s side:

2023-11-04 10:05:13,329 ERROR HandlerThread:16808 [system_info.py:_save_conda():85] Error saving conda packages: [WinError 2] The system cannot find the file specified
Traceback (most recent call last):
File “c:\Users\lscala\AppData\Local\anaconda3\envs\opensoundscape\Lib\site-packages\wandb\sdk\internal\system\system_info.py”, line 81, in _save_conda
subprocess.call(
File “c:\Users\lscala\AppData\Local\anaconda3\envs\opensoundscape\Lib\subprocess.py”, line 389, in call
with Popen(*popenargs, **kwargs) as p:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “c:\Users\lscala\AppData\Local\anaconda3\envs\opensoundscape\Lib\subprocess.py”, line 1026, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File “c:\Users\lscala\AppData\Local\anaconda3\envs\opensoundscape\Lib\subprocess.py”, line 1538, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Could you check your environment variables? This looks similar to the issue that is discussed in this thread.

raphael-sanandres · November 13, 2023, 5:18pm

Hi Lorenzo, since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!

system · January 7, 2024, 10:58pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Wandb: ERROR Run xxjba37s errored: FileNotFoundError(2, 'No such file or directory') W&B Help sweeps , wandb	4	1142	June 5, 2023
Problems about Error Run when using wandb.agent W&B Help wandb , beginner-friendly	5	920	March 26, 2023
NotFoundError() W&B Help sweeps , wandb	3	665	October 31, 2022
NotFoundError() in Sweep after 1 epoch W&B Help sweeps , wandb	5	393	June 17, 2023
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection r W&B Help wandb	7	3976	October 15, 2023

FileNotFoundError: [Errno 2] No such file or directory

Related topics