Pytorch sync_tensorboard help

W&B Help
1

I’m having some issues getting W&B to sync with Tensorboard in PyTorch. According to this issue and the docs, I should initializing SummaryWriter after W&B init possibly using wandb.tensorboard.patch. So far I haven’t been able to get this to work with either torch.utils.tensorboard or tensorboardX and with or without the patch. Not sure if this is a bug or I’m missing something. Thanks.

Windows 11
Python 3.8.8
wandb 0.12.4
torch 1.9.1

 wandb.tensorboard.patch(root_logdir="logs")
 wandb.init(config=hyperparameter_defaults, project=f"ppo_{env_name}_torch", sync_tensorboard=True, save_code=True, name=run_name)
 config = wandb.config
 writer = SummaryWriter(f"logs")
2

Hey there, taking a look!

1 Like
3

Hey there, are you getting an error? If so could you send the stack trace?

4

I don’t get any errors. The code runs and logs to Tensorboard, but nothing gets logged to W&B. Here’s an example run:

dazzling-microwave-39: Weights & Biases

7

Could you share the debug logs for this run? They are in the run directory relative to your script.

8

debug.log

2021-10-26 07:58:59,137 INFO    MainThread:33572 [wandb_setup.py:_flush():71] setting env: {}
2021-10-26 07:58:59,137 INFO    MainThread:33572 [wandb_setup.py:_flush():71] setting login settings: {}
2021-10-26 07:58:59,138 INFO    MainThread:33572 [wandb_init.py:_log_setup():357] Logging user logs to C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\logs\debug.log
2021-10-26 07:58:59,138 INFO    MainThread:33572 [wandb_init.py:_log_setup():358] Logging internal logs to C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\logs\debug-internal.log
2021-10-26 07:58:59,139 INFO    MainThread:33572 [wandb_init.py:init():390] calling init triggers
2021-10-26 07:58:59,139 INFO    MainThread:33572 [wandb_init.py:init():395] wandb.init called with sweep_config: {}
config: {'gamma': 0.99, 'gae_lambda': 0.95, 'eps': 0.2, 'batch_size': 128, 'timesteps': 2000, 'n_epochs': 40, 'c1': 0.5, 'c2': 0.001, 'actor_lr': 0.0003, 'critic_lr': 0.001, 'decay_lr': True, 'lr_decay': 1, 'epochs': 150, 'test_freq': 20, 'reward_threshold': 150}
2021-10-26 07:58:59,139 INFO    MainThread:33572 [wandb_init.py:init():435] starting backend
2021-10-26 07:58:59,139 INFO    MainThread:33572 [backend.py:_multiprocessing_setup():94] multiprocessing start_methods=spawn, using: spawn
2021-10-26 07:58:59,151 INFO    MainThread:33572 [backend.py:ensure_launched():198] starting backend process...
2021-10-26 07:58:59,266 INFO    MainThread:33572 [backend.py:ensure_launched():203] started backend process with pid: 16652
2021-10-26 07:58:59,267 INFO    MainThread:33572 [wandb_init.py:init():444] backend started and connected
2021-10-26 07:58:59,267 INFO    MainThread:33572 [wandb_init.py:init():503] updated telemetry
2021-10-26 07:58:59,346 INFO    MainThread:33572 [wandb_init.py:init():533] communicating current version
2021-10-26 07:59:01,269 INFO    MainThread:33572 [wandb_init.py:init():538] got version response 
2021-10-26 07:59:01,269 INFO    MainThread:33572 [wandb_init.py:init():548] communicating run to backend with 30 second timeout
2021-10-26 07:59:01,362 INFO    MainThread:33572 [wandb_init.py:init():576] starting run threads in backend
2021-10-26 07:59:06,373 INFO    MainThread:33572 [wandb_run.py:_console_start():1693] atexit reg
2021-10-26 07:59:06,374 INFO    MainThread:33572 [wandb_run.py:_redirect():1567] redirect: SettingsConsole.WRAP
2021-10-26 07:59:06,374 INFO    MainThread:33572 [wandb_run.py:_redirect():1604] Wrapping output streams.
2021-10-26 07:59:06,377 INFO    MainThread:33572 [wandb_run.py:_redirect():1628] Redirects installed.
2021-10-26 07:59:06,377 INFO    MainThread:33572 [wandb_init.py:init():603] run started, returning control to user process
2021-10-26 07:59:06,377 INFO    MainThread:33572 [wandb_watch.py:watch():43] Watching
2021-10-26 07:59:06,378 INFO    MainThread:33572 [wandb_run.py:_tensorboard_callback():984] tensorboard callback: logs, None
2021-10-26 08:01:02,573 INFO    MainThread:33572 [wandb_run.py:_atexit_cleanup():1663] got exitcode: 255
2021-10-26 08:01:02,575 INFO    MainThread:33572 [wandb_run.py:_restore():1635] restore
2021-10-26 08:01:02,833 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: file_counts {
  wandb_count: 2
  other_count: 1
}
pusher_stats {
  uploaded_bytes: 41021
  total_bytes: 41021
}

2021-10-26 08:01:07,935 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: None
2021-10-26 08:01:09,929 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: file_counts {
  wandb_count: 2
  other_count: 1
}
pusher_stats {
  uploaded_bytes: 41021
  total_bytes: 41021
}

2021-10-26 08:01:10,043 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: file_counts {
  wandb_count: 2
  other_count: 1
}
pusher_stats {
  uploaded_bytes: 41021
  total_bytes: 41021
}

2021-10-26 08:01:10,560 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: file_counts {
  wandb_count: 6
  other_count: 1
}
pusher_stats {
  uploaded_bytes: 41021
  total_bytes: 66539
}

2021-10-26 08:01:10,667 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: file_counts {
  wandb_count: 7
  other_count: 1
}
pusher_stats {
  uploaded_bytes: 41021
  total_bytes: 88164
}

2021-10-26 08:01:10,769 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: file_counts {
  wandb_count: 7
  other_count: 1
}
pusher_stats {
  uploaded_bytes: 88164
  total_bytes: 88164
}

2021-10-26 08:01:10,871 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: file_counts {
  wandb_count: 7
  other_count: 1
}
pusher_stats {
  uploaded_bytes: 88164
  total_bytes: 88164
}

2021-10-26 08:01:10,972 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: file_counts {
  wandb_count: 7
  other_count: 1
}
pusher_stats {
  uploaded_bytes: 88164
  total_bytes: 88164
}

2021-10-26 08:01:11,104 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: file_counts {
  wandb_count: 7
  other_count: 1
}
pusher_stats {
  uploaded_bytes: 88164
  total_bytes: 88164
}

2021-10-26 08:01:11,319 INFO    MainThread:33572 [wandb_run.py:_wait_for_finish():1793] got exit ret: done: true
exit_result {
}
file_counts {
  wandb_count: 7
  other_count: 1
}
pusher_stats {
  uploaded_bytes: 88164
  total_bytes: 88164
}
local_info {
}

2021-10-26 08:01:12,789 INFO    MainThread:33572 [wandb_run.py:_append_history():2011] rendering history
2021-10-26 08:01:12,789 INFO    MainThread:33572 [wandb_run.py:_append_summary():1966] rendering summary
2021-10-26 08:01:12,789 INFO    MainThread:33572 [wandb_run.py:_append_files():2061] logging synced files
9

debug-internal.log

2021-10-26 07:59:01,207 INFO    MainThread:16652 [internal.py:wandb_internal():88] W&B internal server running at pid: 16652, started at: 2021-10-26 07:59:01.207113
2021-10-26 07:59:01,209 INFO    WriterThread:16652 [datastore.py:open_for_write():77] open: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\run-8x3u2qoi.wandb
2021-10-26 07:59:01,209 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: check_version
2021-10-26 07:59:01,213 DEBUG   SenderThread:16652 [sender.py:send():185] send: header
2021-10-26 07:59:01,213 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: check_version
2021-10-26 07:59:01,270 DEBUG   SenderThread:16652 [sender.py:send():185] send: run
2021-10-26 07:59:01,362 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: run_start
2021-10-26 07:59:01,364 INFO    SenderThread:16652 [dir_watcher.py:__init__():169] watching files in: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files
2021-10-26 07:59:01,364 INFO    SenderThread:16652 [sender.py:_start_run_threads():768] run started: 8x3u2qoi with start time 1635256739
2021-10-26 07:59:01,364 DEBUG   SenderThread:16652 [sender.py:send():185] send: summary
2021-10-26 07:59:01,364 INFO    SenderThread:16652 [sender.py:_save_file():901] saving file wandb-summary.json with policy end
2021-10-26 07:59:02,365 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_created():217] file/dir created: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\wandb-summary.json
2021-10-26 07:59:03,592 DEBUG   HandlerThread:16652 [meta.py:__init__():40] meta init
2021-10-26 07:59:03,593 DEBUG   HandlerThread:16652 [meta.py:__init__():54] meta init done
2021-10-26 07:59:03,593 DEBUG   HandlerThread:16652 [meta.py:probe():212] probe
2021-10-26 07:59:03,645 DEBUG   HandlerThread:16652 [meta.py:_setup_git():202] setup git
2021-10-26 07:59:03,796 DEBUG   HandlerThread:16652 [meta.py:_setup_git():209] setup git done
2021-10-26 07:59:03,796 DEBUG   HandlerThread:16652 [meta.py:_save_code():90] save code
2021-10-26 07:59:03,873 DEBUG   HandlerThread:16652 [meta.py:_save_code():111] save code done
2021-10-26 07:59:03,873 DEBUG   HandlerThread:16652 [meta.py:_save_patches():128] save patches
2021-10-26 07:59:04,310 DEBUG   HandlerThread:16652 [meta.py:_save_patches():170] save patches done
2021-10-26 07:59:04,310 DEBUG   HandlerThread:16652 [meta.py:_save_pip():58] save pip
2021-10-26 07:59:04,311 DEBUG   HandlerThread:16652 [meta.py:_save_pip():72] save pip done
2021-10-26 07:59:04,312 DEBUG   HandlerThread:16652 [meta.py:_save_conda():79] save conda
2021-10-26 07:59:04,364 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_created():217] file/dir created: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\conda-environment.yaml
2021-10-26 07:59:04,365 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_created():217] file/dir created: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\code\ppo_torch.py
2021-10-26 07:59:04,365 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_created():217] file/dir created: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\requirements.txt
2021-10-26 07:59:04,365 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_created():217] file/dir created: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\diff.patch
2021-10-26 07:59:04,365 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_created():217] file/dir created: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\code
2021-10-26 07:59:07,367 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_created():217] file/dir created: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:11,371 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:15,374 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:18,406 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\conda-environment.yaml
2021-10-26 07:59:18,407 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:18,630 DEBUG   HandlerThread:16652 [meta.py:_save_conda():87] save conda done
2021-10-26 07:59:18,631 DEBUG   HandlerThread:16652 [meta.py:probe():250] probe done
2021-10-26 07:59:18,636 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: stop_status
2021-10-26 07:59:18,636 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: stop_status
2021-10-26 07:59:18,638 INFO    HandlerThread:16652 [handler.py:handle_tbrecord():640] handling tbrecord: tbrecord {
  log_dir: "logs"
  save: true
}

2021-10-26 07:59:18,639 DEBUG   HandlerThread:16652 [config_util.py:dict_from_config_file():101] no default config file found in config-defaults.yaml
2021-10-26 07:59:18,667 DEBUG   Thread-30 :16652 [tb_watcher.py:_process_events():269] Encountered tensorboard directory watcher error: [WinError 1314] A required privilege is not held by the client: 'C:\\Users\\tim\\GitHub\\rl_agents\\logs\\events.out.tfevents.1635256746.DESKTOP-7MG5U16.33572.0' -> 'C:\\Users\\tim\\GitHub\\rl_agents\\wandb\\run-20211026_075859-8x3u2qoi\\files\\events.out.tfevents.1635256746.DESKTOP-7MG5U16.33572.0'
2021-10-26 07:59:18,695 DEBUG   SenderThread:16652 [sender.py:send():185] send: tbrecord
2021-10-26 07:59:18,696 DEBUG   SenderThread:16652 [sender.py:send():185] send: files
2021-10-26 07:59:18,696 INFO    SenderThread:16652 [sender.py:_save_file():901] saving file wandb-metadata.json with policy now
2021-10-26 07:59:18,697 INFO    SenderThread:16652 [sender.py:_save_file():901] saving file code\ppo_torch.py with policy now
2021-10-26 07:59:18,699 INFO    SenderThread:16652 [sender.py:_save_file():901] saving file diff.patch with policy now
2021-10-26 07:59:18,987 INFO    Thread-31 :16652 [upload_job.py:push():137] Uploaded file C:\Users\tim\AppData\Local\Temp\tmp70meo4ibwandb\1turuybf-wandb-metadata.json
2021-10-26 07:59:18,999 INFO    Thread-33 :16652 [upload_job.py:push():137] Uploaded file C:\Users\tim\AppData\Local\Temp\tmp70meo4ibwandb\2dtlm77i-diff.patch
2021-10-26 07:59:19,045 INFO    Thread-32 :16652 [upload_job.py:push():137] Uploaded file C:\Users\tim\AppData\Local\Temp\tmp70meo4ibwandb\1pyzaeuj-code/ppo_torch.py
2021-10-26 07:59:19,409 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_created():217] file/dir created: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\wandb-metadata.json
2021-10-26 07:59:22,410 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:24,411 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:26,373 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: stop_status
2021-10-26 07:59:26,373 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: stop_status
2021-10-26 07:59:28,416 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:32,419 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:32,654 DEBUG   SenderThread:16652 [sender.py:send():185] send: stats
2021-10-26 07:59:36,420 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:38,423 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:41,439 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: stop_status
2021-10-26 07:59:41,439 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: stop_status
2021-10-26 07:59:42,425 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:44,427 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:48,431 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:52,435 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:54,441 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 07:59:56,493 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: stop_status
2021-10-26 07:59:56,493 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: stop_status
2021-10-26 07:59:58,444 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:02,453 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:03,625 DEBUG   SenderThread:16652 [sender.py:send():185] send: stats
2021-10-26 08:00:06,457 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:10,458 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:11,551 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: stop_status
2021-10-26 08:00:11,551 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: stop_status
2021-10-26 08:00:12,460 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:16,463 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:18,464 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:22,468 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:24,470 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:26,622 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: stop_status
2021-10-26 08:00:26,622 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: stop_status
2021-10-26 08:00:28,472 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:30,475 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:34,478 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:34,490 DEBUG   SenderThread:16652 [sender.py:send():185] send: stats
2021-10-26 08:00:38,481 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:40,481 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:41,679 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: stop_status
2021-10-26 08:00:41,679 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: stop_status
2021-10-26 08:00:44,485 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:46,489 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:50,492 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:52,493 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:56,496 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:00:56,755 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: stop_status
2021-10-26 08:00:56,755 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: stop_status
2021-10-26 08:01:00,500 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:01:02,500 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:01:02,569 WARNING MainThread:16652 [internal.py:wandb_internal():147] Internal process interrupt: 1
2021-10-26 08:01:02,578 DEBUG   SenderThread:16652 [sender.py:send():185] send: history
2021-10-26 08:01:02,579 DEBUG   SenderThread:16652 [sender.py:send():185] send: summary
2021-10-26 08:01:02,582 INFO    SenderThread:16652 [sender.py:_save_file():901] saving file wandb-summary.json with policy end
2021-10-26 08:01:02,828 DEBUG   SenderThread:16652 [sender.py:send():185] send: telemetry
2021-10-26 08:01:02,829 DEBUG   SenderThread:16652 [sender.py:send():185] send: exit
2021-10-26 08:01:02,829 INFO    SenderThread:16652 [sender.py:send_exit():317] handling exit code: 255
2021-10-26 08:01:02,829 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:02,829 INFO    SenderThread:16652 [sender.py:send_exit():319] handling runtime: 121
2021-10-26 08:01:02,831 INFO    SenderThread:16652 [sender.py:_save_file():901] saving file wandb-summary.json with policy end
2021-10-26 08:01:02,832 INFO    SenderThread:16652 [sender.py:send_exit():329] send defer
2021-10-26 08:01:02,832 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:02,832 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: defer
2021-10-26 08:01:02,833 INFO    HandlerThread:16652 [handler.py:handle_request_defer():147] handle defer: 0
2021-10-26 08:01:02,833 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: defer
2021-10-26 08:01:02,833 INFO    SenderThread:16652 [sender.py:send_request_defer():338] handle sender defer: 0
2021-10-26 08:01:02,833 INFO    SenderThread:16652 [sender.py:transition_state():342] send defer: 1
2021-10-26 08:01:02,833 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: defer
2021-10-26 08:01:02,833 INFO    HandlerThread:16652 [handler.py:handle_request_defer():147] handle defer: 1
2021-10-26 08:01:02,923 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: defer
2021-10-26 08:01:02,923 INFO    SenderThread:16652 [sender.py:send_request_defer():338] handle sender defer: 1
2021-10-26 08:01:02,923 INFO    SenderThread:16652 [sender.py:transition_state():342] send defer: 2
2021-10-26 08:01:02,924 DEBUG   SenderThread:16652 [sender.py:send():185] send: stats
2021-10-26 08:01:02,924 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: defer
2021-10-26 08:01:02,925 INFO    HandlerThread:16652 [handler.py:handle_request_defer():147] handle defer: 2
2021-10-26 08:01:03,502 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\wandb-summary.json
2021-10-26 08:01:03,503 INFO    Thread-12 :16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:01:09,927 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: defer
2021-10-26 08:01:09,927 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:09,927 INFO    SenderThread:16652 [sender.py:send_request_defer():338] handle sender defer: 2
2021-10-26 08:01:09,927 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:09,928 INFO    SenderThread:16652 [sender.py:transition_state():342] send defer: 3
2021-10-26 08:01:09,928 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:09,928 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:09,928 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: defer
2021-10-26 08:01:09,929 INFO    HandlerThread:16652 [handler.py:handle_request_defer():147] handle defer: 3
2021-10-26 08:01:09,930 DEBUG   SenderThread:16652 [sender.py:send():185] send: summary
2021-10-26 08:01:09,934 INFO    SenderThread:16652 [sender.py:_save_file():901] saving file wandb-summary.json with policy end
2021-10-26 08:01:09,934 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: defer
2021-10-26 08:01:09,934 INFO    SenderThread:16652 [sender.py:send_request_defer():338] handle sender defer: 3
2021-10-26 08:01:09,934 INFO    SenderThread:16652 [sender.py:transition_state():342] send defer: 4
2021-10-26 08:01:09,935 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: defer
2021-10-26 08:01:09,936 INFO    HandlerThread:16652 [handler.py:handle_request_defer():147] handle defer: 4
2021-10-26 08:01:09,936 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: defer
2021-10-26 08:01:09,936 INFO    SenderThread:16652 [sender.py:send_request_defer():338] handle sender defer: 4
2021-10-26 08:01:10,030 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:10,042 INFO    SenderThread:16652 [sender.py:transition_state():342] send defer: 5
2021-10-26 08:01:10,043 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:10,043 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: defer
2021-10-26 08:01:10,043 INFO    HandlerThread:16652 [handler.py:handle_request_defer():147] handle defer: 5
2021-10-26 08:01:10,044 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: defer
2021-10-26 08:01:10,044 INFO    SenderThread:16652 [sender.py:send_request_defer():338] handle sender defer: 5
2021-10-26 08:01:10,044 INFO    SenderThread:16652 [dir_watcher.py:finish():283] shutting down directory watcher
2021-10-26 08:01:10,145 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:10,509 INFO    SenderThread:16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\wandb-summary.json
2021-10-26 08:01:10,510 INFO    SenderThread:16652 [dir_watcher.py:_on_file_modified():230] file/dir modified: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\config.yaml
2021-10-26 08:01:10,512 INFO    SenderThread:16652 [dir_watcher.py:finish():313] scan: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files
2021-10-26 08:01:10,513 INFO    SenderThread:16652 [dir_watcher.py:finish():327] scan save: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\conda-environment.yaml conda-environment.yaml
2021-10-26 08:01:10,515 INFO    SenderThread:16652 [dir_watcher.py:finish():327] scan save: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\config.yaml config.yaml
2021-10-26 08:01:10,517 INFO    SenderThread:16652 [dir_watcher.py:finish():327] scan save: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\diff.patch diff.patch
2021-10-26 08:01:10,518 INFO    SenderThread:16652 [dir_watcher.py:finish():327] scan save: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log output.log
2021-10-26 08:01:10,535 INFO    SenderThread:16652 [dir_watcher.py:finish():327] scan save: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\requirements.txt requirements.txt
2021-10-26 08:01:10,537 INFO    SenderThread:16652 [dir_watcher.py:finish():327] scan save: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\wandb-metadata.json wandb-metadata.json
2021-10-26 08:01:10,537 INFO    SenderThread:16652 [dir_watcher.py:finish():327] scan save: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\wandb-summary.json wandb-summary.json
2021-10-26 08:01:10,553 INFO    SenderThread:16652 [dir_watcher.py:finish():327] scan save: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\code\ppo_torch.py code\ppo_torch.py
2021-10-26 08:01:10,553 INFO    SenderThread:16652 [sender.py:transition_state():342] send defer: 6
2021-10-26 08:01:10,554 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:10,556 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: defer
2021-10-26 08:01:10,557 INFO    HandlerThread:16652 [handler.py:handle_request_defer():147] handle defer: 6
2021-10-26 08:01:10,559 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: defer
2021-10-26 08:01:10,559 INFO    SenderThread:16652 [sender.py:send_request_defer():338] handle sender defer: 6
2021-10-26 08:01:10,559 INFO    SenderThread:16652 [file_pusher.py:finish():177] shutting down file pusher
2021-10-26 08:01:10,666 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:10,666 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:10,768 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:10,768 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:10,804 INFO    Thread-37 :16652 [upload_job.py:push():137] Uploaded file C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\requirements.txt
2021-10-26 08:01:10,842 INFO    Thread-34 :16652 [upload_job.py:push():137] Uploaded file C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\conda-environment.yaml
2021-10-26 08:01:10,856 INFO    Thread-36 :16652 [upload_job.py:push():137] Uploaded file C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\output.log
2021-10-26 08:01:10,856 INFO    Thread-35 :16652 [upload_job.py:push():137] Uploaded file C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\config.yaml
2021-10-26 08:01:10,856 INFO    Thread-38 :16652 [upload_job.py:push():137] Uploaded file C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\files\wandb-summary.json
2021-10-26 08:01:10,870 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:10,870 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:10,971 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:10,971 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:11,057 INFO    Thread-11 :16652 [sender.py:transition_state():342] send defer: 7
2021-10-26 08:01:11,057 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: defer
2021-10-26 08:01:11,057 INFO    HandlerThread:16652 [handler.py:handle_request_defer():147] handle defer: 7
2021-10-26 08:01:11,057 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: defer
2021-10-26 08:01:11,058 INFO    SenderThread:16652 [sender.py:send_request_defer():338] handle sender defer: 7
2021-10-26 08:01:11,073 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:11,103 INFO    SenderThread:16652 [sender.py:transition_state():342] send defer: 8
2021-10-26 08:01:11,103 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:11,104 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: defer
2021-10-26 08:01:11,104 INFO    HandlerThread:16652 [handler.py:handle_request_defer():147] handle defer: 8
2021-10-26 08:01:11,104 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: defer
2021-10-26 08:01:11,104 INFO    SenderThread:16652 [sender.py:send_request_defer():338] handle sender defer: 8
2021-10-26 08:01:11,104 INFO    SenderThread:16652 [sender.py:transition_state():342] send defer: 9
2021-10-26 08:01:11,105 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: defer
2021-10-26 08:01:11,105 INFO    HandlerThread:16652 [handler.py:handle_request_defer():147] handle defer: 9
2021-10-26 08:01:11,105 DEBUG   SenderThread:16652 [sender.py:send():185] send: final
2021-10-26 08:01:11,106 DEBUG   SenderThread:16652 [sender.py:send():185] send: footer
2021-10-26 08:01:11,106 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: defer
2021-10-26 08:01:11,106 INFO    SenderThread:16652 [sender.py:send_request_defer():338] handle sender defer: 9
2021-10-26 08:01:11,205 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: poll_exit
2021-10-26 08:01:11,205 DEBUG   SenderThread:16652 [sender.py:send_request():199] send_request: poll_exit
2021-10-26 08:01:11,205 INFO    SenderThread:16652 [file_pusher.py:join():182] waiting for file pusher
2021-10-26 08:01:11,321 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: get_summary
2021-10-26 08:01:11,324 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: sampled_history
2021-10-26 08:01:11,324 DEBUG   HandlerThread:16652 [handler.py:handle_request():130] handle_request: shutdown
2021-10-26 08:01:11,325 INFO    HandlerThread:16652 [handler.py:finish():731] shutting down handler
2021-10-26 08:01:12,106 INFO    WriterThread:16652 [datastore.py:close():281] close: C:\Users\tim\GitHub\rl_agents\wandb\run-20211026_075859-8x3u2qoi\run-8x3u2qoi.wandb
2021-10-26 08:01:12,320 INFO    SenderThread:16652 [sender.py:finish():1029] shutting down sender
2021-10-26 08:01:12,320 INFO    SenderThread:16652 [file_pusher.py:finish():177] shutting down file pusher
2021-10-26 08:01:12,320 INFO    SenderThread:16652 [file_pusher.py:join():182] waiting for file pusher
2021-10-26 08:01:12,322 INFO    MainThread:16652 [internal.py:handle_exit():78] Internal process exited
10

@tims457 I am terribly sorry for the delay. Removing the sync_tensorboard=True argument from wandb.init(). If you want more information on using wandb.tensorboard.patch() check out this FAQ

11

Not sure if you’ve solved this by now. I’ve expereinced something similar. It turned out that wandb wants to create a symbolic link which it can’t simply do on Windows. When I run my script as admin everything worls fine. You could try that as well.
It’s not very convenient but works for now.