Pytorch sync_tensorboard help

I’m having some issues getting W&B to sync with Tensorboard in PyTorch. According to this issue and the docs, I should initializing SummaryWriter after W&B init possibly using wandb.tensorboard.patch. So far I haven’t been able to get this to work with either torch.utils.tensorboard or tensorboardX and with or without the patch. Not sure if this is a bug or I’m missing something. Thanks.

Windows 11
Python 3.8.8
wandb 0.12.4
torch 1.9.1

 wandb.tensorboard.patch(root_logdir="logs")
 wandb.init(config=hyperparameter_defaults, project=f"ppo_{env_name}_torch", sync_tensorboard=True, save_code=True, name=run_name)
 config = wandb.config
 writer = SummaryWriter(f"logs")

Hey there, taking a look!

1 Like

Hey there, are you getting an error? If so could you send the stack trace?

I don’t get any errors. The code runs and logs to Tensorboard, but nothing gets logged to W&B. Here’s an example run:

dazzling-microwave-39: Weights & Biases

Could you share the debug logs for this run? They are in the run directory relative to your script.

debug.log

2021-10-26 07:58:59,137 INFO    MainThread:33572 [wandb_setup.py:_flush():71] setting env: {}
2021-10-26 07:58:59,137 INFO    MainThread:33572 [wandb_setup.py:_flush():71] setting login settings: {}
2021-10-26 07:58:59,138 INFO    MainThread:33572 [wandb_init.py:_log_setup():357] Logging user logs to C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\logs\debug.log
2021-10-26 07:58:59,138 INFO    MainThread:33572 [wandb_init.py:_log_setup():358] Logging internal logs to C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\logs\debug-internal.log
2021-10-26 07:58:59,139 INFO    MainThread:33572 [wandb_init.py:init():390] calling init triggers
2021-10-26 07:58:59,139 INFO    MainThread:33572 [wandb_init.py:init():395] wandb.init called with sweep_config: {}
config: {'gamma': 0.99, 'gae_lambda': 0.95, 'eps': 0.2, 'batch_size': 128, 'timesteps': 2000, 'n_epochs': 40, 'c1': 0.5, 'c2': 0.001, 'actor_lr': 0.0003, 'critic_lr': 0.001, 'decay_lr': True, 'lr_decay': 1, 'epochs': 150, 'test_freq': 20, 'reward_threshold': 150}
2021-10-26 07:58:59,139 INFO    MainThread:33572 [wandb_init.py:init():435] starting backend
2021-10-26 07:58:59,139 INFO    MainThread:33572 [backend.py:_multiprocessing_setup():94] multiprocessing start_methods=spawn, using: spawn
2021-10-26 07:58:59,151 INFO    MainThread:33572 [backend.py:ensure_launched():198] starting backend process...
2021-10-26 07:58:59,266 INFO    MainThread:33572 [backend.py:ensure_launched():203] started backend process with pid: 16652
2021-10-26 07:58:59,267 INFO    MainThread:33572 [wandb_init.py:init():444] backend started and connected
2021-10-26 07:58:59,267 INFO    MainThread:33572 [wandb_init.py:init():503] updated telemetry
2021-10-26 07:58:59,346 INFO    MainThread:33572 [wandb_init.py:init():533] communicating current version
2021-10-26 07:59:01,269 INFO    MainThread:33572 [wandb_init.py:init():538] got version response 
2021-10-26 07:59:01,269 INFO    MainThread:33572 [wandb_init.py:init():548] communicating run to backend with 30 second timeout
2021-10-26 07:59:01,362 INFO    MainThread:33572 [wandb_init.py:init():576] starting run threads in backend
2021-10-26 07:59:06,373 INFO    MainThread:33572 [wandb_run.py:_console_start():1693] atexit reg
2021-10-26 07:59:06,374 INFO    MainThread:33572 [wandb_run.py:_redirect():1567] redirect: SettingsConsole.WRAP
2021-10-26 07:59:06,374 INFO    MainThread:33572 [wandb_run.py:_redirect():1604] Wrapping output streams.
2021-10-26 07:59:06,377 INFO    MainThread:33572 [wandb_run.py:_redirect():1628] Redirects installed.
2021-10-26 07:59:06,377 INFO    MainThread:33572 [wandb_init.py:init():603] run started, returning control to user process
2021-10-26 07:59:06,377 INFO    MainThread:33572 [wandb_watch.py:watch():43] Watching
2021-10-26 07:59:06,378 INFO    MainThread:33572 [wandb_run.py:_tensorboard_callback():984] tensorboard callback: logs, None
2021-10-26 08:01:02,573 INFO    MainThread:33572 [wandb_run.py:_atexit_cleanup():1663] got exitcode: 255
2021-10-26 08:01:02,575 INFO    MainThread:33572 [wandb_run.py:_restore():1635] restore
2021-10-26 08:01:02,833 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: file_counts {
  wandb_count: 2
  other_count: 1
}
pusher_stats {
  uploaded_bytes: 41021
  total_bytes: 41021
}

2021-10-26 08:01:07,935 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: None
2021-10-26 08:01:09,929 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: file_counts {
  wandb_count: 2
  other_count: 1
}
pusher_stats {
  uploaded_bytes: 41021
  total_bytes: 41021
}

2021-10-26 08:01:10,043 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: file_counts {
  wandb_count: 2
  other_count: 1
}
pusher_stats {
  uploaded_bytes: 41021
  total_bytes: 41021
}

2021-10-26 08:01:10,560 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: file_counts {
  wandb_count: 6
  other_count: 1
}
pusher_stats {
  uploaded_bytes: 41021
  total_bytes: 66539
}

2021-10-26 08:01:10,667 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: file_counts {
  wandb_count: 7
  other_count: 1
}
pusher_stats {
  uploaded_bytes: 41021
  total_bytes: 88164
}

2021-10-26 08:01:10,769 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: file_counts {
  wandb_count: 7
  other_count: 1
}
pusher_stats {
  uploaded_bytes: 88164
  total_bytes: 88164
}

2021-10-26 08:01:10,871 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: file_counts {
  wandb_count: 7
  other_count: 1
}
pusher_stats {
  uploaded_bytes: 88164
  total_bytes: 88164
}

2021-10-26 08:01:10,972 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: file_counts {
  wandb_count: 7
  other_count: 1
}
pusher_stats {
  uploaded_bytes: 88164
  total_bytes: 88164
}

2021-10-26 08:01:11,104 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: file_counts {
  wandb_count: 7
  other_count: 1
}
pusher_stats {
  uploaded_bytes: 88164
  total_bytes: 88164
}

2021-10-26 08:01:11,319 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: done: true
exit_result {
}
file_counts {
  wandb_count: 7
  other_count: 1
}
pusher_stats {
  uploaded_bytes: 88164
  total_bytes: 88164
}
local_info {
}

2021-10-26 08:01:12,789 INFO    MainThread:33572 [wandb_run.py:_append_history():2011] rendering history
2021-10-26 08:01:12,789 INFO    MainThread:33572 [wandb_run.py:_append_summary():1966] rendering summary
2021-10-26 08:01:12,789 INFO    MainThread:33572 [wandb_run.py:_append_files():2061] logging synced files

debug-internal.log

2021-10-26 07:59:01,207 INFO    MainThread:16652 [internal.py:wandb_internal():88] W&B internal server running at pid: 16652, started at: 2021-10-26 07:59:01.207113
2021-10-26 07:59:01,209 INFO    WriterThread:16652 [datastore.py:open_for_write():77] open: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\run-8x3u2qoi.wandb
2021-10-26 07:59:01,209 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: check_version
2021-10-26 07:59:01,213 DEBUG   SenderThread:16652 [sender.py:send():185] send: header
2021-10-26 07:59:01,213 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: check_version
2021-10-26 07:59:01,270 DEBUG   SenderThread:16652 [sender.py:send():185] send: run
2021-10-26 07:59:01,362 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: run_start
2021-10-26 07:59:01,364 INFO    SenderThread:16652 [dir_watcher.py:__init__():169] watching files in: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files
2021-10-26 07:59:01,364 INFO    SenderThread:16652 [sender.py:_start_run_threads():768] run started: 8x3u2qoi with start time 1635256739
2021-10-26 07:59:01,364 DEBUG   SenderThread:16652 [sender.py:send():185] send: summary
2021-10-26 07:59:01,364 INFO    SenderThread:16652 [sender.py:_save_file():901] saving file wandb-summary.json with policy end
2021-10-26 07:59:02,365 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_created():217] file/dir created: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\wandb-summary.json
2021-10-26 07:59:03,592 DEBUG   HandlerThread:16652 [meta.py:__init__():40] meta init
2021-10-26 07:59:03,593 DEBUG   HandlerThread:16652 [meta.py:__init__():54] meta init done
2021-10-26 07:59:03,593 DEBUG   HandlerThread:16652 [meta.py:probe():212] probe
2021-10-26 07:59:03,645 DEBUG   HandlerThread:16652 [meta.py:_setup_git():202] setup git
2021-10-26 07:59:03,796 DEBUG   HandlerThread:16652 [meta.py:_setup_git():209] setup git done
2021-10-26 07:59:03,796 DEBUG   HandlerThread:16652 [meta.py:_save_code():90] save code
2021-10-26 07:59:03,873 DEBUG   HandlerThread:16652 [meta.py:_save_code():111] save code done
2021-10-26 07:59:03,873 DEBUG   HandlerThread:16652 [meta.py:_save_patches():128] save patches
2021-10-26 07:59:04,310 DEBUG   HandlerThread:16652 [meta.py:_save_patches():170] save patches done
2021-10-26 07:59:04,310 DEBUG   HandlerThread:16652 [meta.py:_save_pip():58] save pip
2021-10-26 07:59:04,311 DEBUG   HandlerThread:16652 [meta.py:_save_pip():72] save pip done
2021-10-26 07:59:04,312 DEBUG   HandlerThread:16652 [meta.py:_save_conda():79] save conda
2021-10-26 07:59:04,364 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_created():217] file/dir created: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\conda-environment.yaml
2021-10-26 07:59:04,365 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_created():217] file/dir created: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\code\ppo_torch.py
2021-10-26 07:59:04,365 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_created():217] file/dir created: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\requirements.txt
2021-10-26 07:59:04,365 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_created():217] file/dir created: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\diff.patch
2021-10-26 07:59:04,365 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_created():217] file/dir created: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\code
2021-10-26 07:59:07,367 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_created():217] file/dir created: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:11,371 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:15,374 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:18,406 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\conda-environment.yaml
2021-10-26 07:59:18,407 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:18,630 DEBUG   HandlerThread:16652 [meta.py:_save_conda():87] save conda done
2021-10-26 07:59:18,631 DEBUG   HandlerThread:16652 [meta.py:probe():250] probe done
2021-10-26 07:59:18,636 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: stop_status
2021-10-26 07:59:18,636 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: stop_status
2021-10-26 07:59:18,638 INFO    HandlerThread:16652 [handler.py:handle_tbrecord():640] handling tbrecord: tbrecord {
  log_dir: "logs"
  save: true
}

2021-10-26 07:59:18,639 DEBUG   HandlerThread:16652 [config_util.py:dict_from_config_file():101] no default config file found in config-defaults.yaml
2021-10-26 07:59:18,667 DEBUG   Thread-30 :16652 [tb_watcher.py:_process_events():269] Encountered tensorboard directory watcher error: [WinError 1314] A required privilege is not held by the client: 'C:\\Users\\tim\\GitHub\\rl_agents\\logs\\events.out.tfevents.1635256746.DESKTOP-7MG5U16.33572.0' -> 'C:\\Users\\tim\\GitHub\\rl_agents\\wandb\\run-20211026_075859-8x3u2qoi\\files\\events.out.tfevents.1635256746.DESKTOP-7MG5U16.33572.0'
2021-10-26 07:59:18,695 DEBUG   SenderThread:16652 [sender.py:send():185] send: tbrecord
2021-10-26 07:59:18,696 DEBUG   SenderThread:16652 [sender.py:send():185] send: files
2021-10-26 07:59:18,696 INFO    SenderThread:16652 [sender.py:_save_file():901] saving file wandb-metadata.json with policy now
2021-10-26 07:59:18,697 INFO    SenderThread:16652 [sender.py:_save_file():901] saving file code\ppo_torch.py with policy now
2021-10-26 07:59:18,699 INFO    SenderThread:16652 [sender.py:_save_file():901] saving file diff.patch with policy now
2021-10-26 07:59:18,987 INFO    Thread-31 :16652 [upload_job.py:push():137] Uploaded file C:\Users\tim\AppData\Local\Temp\tmp70meo4ibwandb\1turuybf-wandb-metadata.json
2021-10-26 07:59:18,999 INFO    Thread-33 :16652 [upload_job.py:push():137] Uploaded file C:\Users\tim\AppData\Local\Temp\tmp70meo4ibwandb\2dtlm77i-diff.patch
2021-10-26 07:59:19,045 INFO    Thread-32 :16652 [upload_job.py:push():137] Uploaded file C:\Users\tim\AppData\Local\Temp\tmp70meo4ibwandb\1pyzaeuj-code/ppo_torch.py
2021-10-26 07:59:19,409 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_created():217] file/dir created: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\wandb-metadata.json
2021-10-26 07:59:22,410 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:24,411 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:26,373 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: stop_status
2021-10-26 07:59:26,373 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: stop_status
2021-10-26 07:59:28,416 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:32,419 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:32,654 DEBUG   SenderThread:16652 [sender.py:send():185] send: stats
2021-10-26 07:59:36,420 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:38,423 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:41,439 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: stop_status
2021-10-26 07:59:41,439 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: stop_status
2021-10-26 07:59:42,425 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:44,427 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:48,431 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:52,435 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:54,441 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:56,493 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: stop_status
2021-10-26 07:59:56,493 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: stop_status
2021-10-26 07:59:58,444 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:02,453 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:03,625 DEBUG   SenderThread:16652 [sender.py:send():185] send: stats
2021-10-26 08:00:06,457 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:10,458 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:11,551 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: stop_status
2021-10-26 08:00:11,551 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: stop_status
2021-10-26 08:00:12,460 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:16,463 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:18,464 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:22,468 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:24,470 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:26,622 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: stop_status
2021-10-26 08:00:26,622 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: stop_status
2021-10-26 08:00:28,472 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:30,475 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:34,478 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:34,490 DEBUG   SenderThread:16652 [sender.py:send():185] send: stats
2021-10-26 08:00:38,481 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:40,481 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:41,679 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: stop_status
2021-10-26 08:00:41,679 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: stop_status
2021-10-26 08:00:44,485 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:46,489 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:50,492 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:52,493 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:56,496 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:56,755 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: stop_status
2021-10-26 08:00:56,755 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: stop_status
2021-10-26 08:01:00,500 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:01:02,500 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:01:02,569 WARNING MainThread:16652 [internal.py:wandb_internal():147] Internal process interrupt: 1
2021-10-26 08:01:02,578 DEBUG   SenderThread:16652 [sender.py:send():185] send: history
2021-10-26 08:01:02,579 DEBUG   SenderThread:16652 [sender.py:send():185] send: summary
2021-10-26 08:01:02,582 INFO    SenderThread:16652 [sender.py:_save_file():901] saving file wandb-summary.json with policy end
2021-10-26 08:01:02,828 DEBUG   SenderThread:16652 [sender.py:send():185] send: telemetry
2021-10-26 08:01:02,829 DEBUG   SenderThread:16652 [sender.py:send():185] send: exit
2021-10-26 08:01:02,829 INFO    SenderThread:16652 [sender.py:send_exit():317] handling exit code: 255
2021-10-26 08:01:02,829 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:02,829 INFO    SenderThread:16652 [sender.py:send_exit():319] handling runtime: 121
2021-10-26 08:01:02,831 INFO    SenderThread:16652 [sender.py:_save_file():901] saving file wandb-summary.json with policy end
2021-10-26 08:01:02,832 INFO    SenderThread:16652 [sender.py:send_exit():329] send defer
2021-10-26 08:01:02,832 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:02,832 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: defer
2021-10-26 08:01:02,833 INFO    HandlerThread:16652 [handler.py:handle_request_defer():147] handle defer: 0
2021-10-26 08:01:02,833 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: defer
2021-10-26 08:01:02,833 INFO    SenderThread:16652 [sender.py:send_request_defer():338] handle sender defer: 0
2021-10-26 08:01:02,833 INFO    SenderThread:16652 [sender.py:transition_state():342] send defer: 1
2021-10-26 08:01:02,833 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: defer
2021-10-26 08:01:02,833 INFO    HandlerThread:16652 [handler.py:handle_request_defer():147] handle defer: 1
2021-10-26 08:01:02,923 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: defer
2021-10-26 08:01:02,923 INFO    SenderThread:16652 [sender.py:send_request_defer():338] handle sender defer: 1
2021-10-26 08:01:02,923 INFO    SenderThread:16652 [sender.py:transition_state():342] send defer: 2
2021-10-26 08:01:02,924 DEBUG   SenderThread:16652 [sender.py:send():185] send: stats
2021-10-26 08:01:02,924 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: defer
2021-10-26 08:01:02,925 INFO    HandlerThread:16652 [handler.py:handle_request_defer():147] handle defer: 2
2021-10-26 08:01:03,502 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\wandb-summary.json
2021-10-26 08:01:03,503 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:01:09,927 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: defer
2021-10-26 08:01:09,927 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:09,927 INFO    SenderThread:16652 [sender.py:send_request_defer():338] handle sender defer: 2
2021-10-26 08:01:09,927 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:09,928 INFO    SenderThread:16652 [sender.py:transition_state():342] send defer: 3
2021-10-26 08:01:09,928 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:09,928 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:09,928 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: defer
2021-10-26 08:01:09,929 INFO    HandlerThread:16652 [handler.py:handle_request_defer():147] handle defer: 3
2021-10-26 08:01:09,930 DEBUG   SenderThread:16652 [sender.py:send():185] send: summary
2021-10-26 08:01:09,934 INFO    SenderThread:16652 [sender.py:_save_file():901] saving file wandb-summary.json with policy end
2021-10-26 08:01:09,934 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: defer
2021-10-26 08:01:09,934 INFO    SenderThread:16652 [sender.py:send_request_defer():338] handle sender defer: 3
2021-10-26 08:01:09,934 INFO    SenderThread:16652 [sender.py:transition_state():342] send defer: 4
2021-10-26 08:01:09,935 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: defer
2021-10-26 08:01:09,936 INFO    HandlerThread:16652 [handler.py:handle_request_defer():147] handle defer: 4
2021-10-26 08:01:09,936 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: defer
2021-10-26 08:01:09,936 INFO    SenderThread:16652 [sender.py:send_request_defer():338] handle sender defer: 4
2021-10-26 08:01:10,030 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:10,042 INFO    SenderThread:16652 [sender.py:transition_state():342] send defer: 5
2021-10-26 08:01:10,043 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:10,043 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: defer
2021-10-26 08:01:10,043 INFO    HandlerThread:16652 [handler.py:handle_request_defer():147] handle defer: 5
2021-10-26 08:01:10,044 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: defer
2021-10-26 08:01:10,044 INFO    SenderThread:16652 [sender.py:send_request_defer():338] handle sender defer: 5
2021-10-26 08:01:10,044 INFO    SenderThread:16652 [dir_watcher.py:finish():283] shutting down directory watcher
2021-10-26 08:01:10,145 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:10,509 INFO    SenderThread:16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\wandb-summary.json
2021-10-26 08:01:10,510 INFO    SenderThread:16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\config.yaml
2021-10-26 08:01:10,512 INFO    SenderThread:16652 [dir_watcher.py:finish():313] scan: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files
2021-10-26 08:01:10,513 INFO    SenderThread:16652 [dir_watcher.py:finish():327] scan save: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\conda-environment.yaml conda-environment.yaml
2021-10-26 08:01:10,515 INFO    SenderThread:16652 [dir_watcher.py:finish():327] scan save: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\config.yaml config.yaml
2021-10-26 08:01:10,517 INFO    SenderThread:16652 [dir_watcher.py:finish():327] scan save: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\diff.patch diff.patch
2021-10-26 08:01:10,518 INFO    SenderThread:16652 [dir_watcher.py:finish():327] scan save: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log output.log
2021-10-26 08:01:10,535 INFO    SenderThread:16652 [dir_watcher.py:finish():327] scan save: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\requirements.txt requirements.txt
2021-10-26 08:01:10,537 INFO    SenderThread:16652 [dir_watcher.py:finish():327] scan save: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\wandb-metadata.json wandb-metadata.json
2021-10-26 08:01:10,537 INFO    SenderThread:16652 [dir_watcher.py:finish():327] scan save: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\wandb-summary.json wandb-summary.json
2021-10-26 08:01:10,553 INFO    SenderThread:16652 [dir_watcher.py:finish():327] scan save: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\code\ppo_torch.py code\ppo_torch.py
2021-10-26 08:01:10,553 INFO    SenderThread:16652 [sender.py:transition_state():342] send defer: 6
2021-10-26 08:01:10,554 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:10,556 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: defer
2021-10-26 08:01:10,557 INFO    HandlerThread:16652 [handler.py:handle_request_defer():147] handle defer: 6
2021-10-26 08:01:10,559 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: defer
2021-10-26 08:01:10,559 INFO    SenderThread:16652 [sender.py:send_request_defer():338] handle sender defer: 6
2021-10-26 08:01:10,559 INFO    SenderThread:16652 [file_pusher.py:finish():177] shutting down file pusher
2021-10-26 08:01:10,666 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:10,666 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:10,768 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:10,768 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:10,804 INFO    Thread-37 :16652 [upload_job.py:push():137] Uploaded file C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\requirements.txt
2021-10-26 08:01:10,842 INFO    Thread-34 :16652 [upload_job.py:push():137] Uploaded file C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\conda-environment.yaml
2021-10-26 08:01:10,856 INFO    Thread-36 :16652 [upload_job.py:push():137] Uploaded file C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:01:10,856 INFO    Thread-35 :16652 [upload_job.py:push():137] Uploaded file C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\config.yaml
2021-10-26 08:01:10,856 INFO    Thread-38 :16652 [upload_job.py:push():137] Uploaded file C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\wandb-summary.json
2021-10-26 08:01:10,870 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:10,870 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:10,971 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:10,971 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:11,057 INFO    Thread-11 :16652 [sender.py:transition_state():342] send defer: 7
2021-10-26 08:01:11,057 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: defer
2021-10-26 08:01:11,057 INFO    HandlerThread:16652 [handler.py:handle_request_defer():147] handle defer: 7
2021-10-26 08:01:11,057 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: defer
2021-10-26 08:01:11,058 INFO    SenderThread:16652 [sender.py:send_request_defer():338] handle sender defer: 7
2021-10-26 08:01:11,073 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:11,103 INFO    SenderThread:16652 [sender.py:transition_state():342] send defer: 8
2021-10-26 08:01:11,103 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:11,104 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: defer
2021-10-26 08:01:11,104 INFO    HandlerThread:16652 [handler.py:handle_request_defer():147] handle defer: 8
2021-10-26 08:01:11,104 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: defer
2021-10-26 08:01:11,104 INFO    SenderThread:16652 [sender.py:send_request_defer():338] handle sender defer: 8
2021-10-26 08:01:11,104 INFO    SenderThread:16652 [sender.py:transition_state():342] send defer: 9
2021-10-26 08:01:11,105 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: defer
2021-10-26 08:01:11,105 INFO    HandlerThread:16652 [handler.py:handle_request_defer():147] handle defer: 9
2021-10-26 08:01:11,105 DEBUG   SenderThread:16652 [sender.py:send():185] send: final
2021-10-26 08:01:11,106 DEBUG   SenderThread:16652 [sender.py:send():185] send: footer
2021-10-26 08:01:11,106 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: defer
2021-10-26 08:01:11,106 INFO    SenderThread:16652 [sender.py:send_request_defer():338] handle sender defer: 9
2021-10-26 08:01:11,205 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:11,205 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:11,205 INFO    SenderThread:16652 [file_pusher.py:join():182] waiting for file pusher
2021-10-26 08:01:11,321 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: get_summary
2021-10-26 08:01:11,324 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: sampled_history
2021-10-26 08:01:11,324 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: shutdown
2021-10-26 08:01:11,325 INFO    HandlerThread:16652 [handler.py:finish():731] shutting down handler
2021-10-26 08:01:12,106 INFO    WriterThread:16652 [datastore.py:close():281] close: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\run-8x3u2qoi.wandb
2021-10-26 08:01:12,320 INFO    SenderThread:16652 [sender.py:finish():1029] shutting down sender
2021-10-26 08:01:12,320 INFO    SenderThread:16652 [file_pusher.py:finish():177] shutting down file pusher
2021-10-26 08:01:12,320 INFO    SenderThread:16652 [file_pusher.py:join():182] waiting for file pusher
2021-10-26 08:01:12,322 INFO    MainThread:16652 [internal.py:handle_exit():78] Internal process exited