Hi,
for our team-project I just added logging images, but they do not appear in the online-view.
What I can see is the following:
If I look at the network traffic, I can see that all the image files have size 0. Locally, in the artifacts folder, I can see proper .png image files, so it seems they are not uploaded.
During the training, we’re logging using the simple wandb.log({'loss': loss, 'image1': wandb.Image(...), 'image2 ': wandb.Image(...)}) function, no additional arguments.
Interestingly, we get images in a follow-up run, but not in the first run. Any idea what may cause this?
Hi @lvoegtle thank you for reporting this issue. Could you please share a link to your Workspace so we can investigate further? Additionally, we need the SDK debug files debug.log and debug-internal.log from the run that logged those images. These debug files are located in your local folder wandb/run-<date>_<time>-<run-id>/logs in the same directory where you’re running your code.
If you prefer not to share these details in our public forum, please email them to us at support@wandb.com and refer to ticket #69516. Thank you!
Hi @thanos-wandb,
thanks for your reply. There are no log-files and no log-folder in the wandb-run folder for the first run. All follow-up runs have that log folder (only “files” containing the images that are missing). In the follow up-folder logs though, I can see that we’re not running on the latest wandb library. We’ll update the container and check if it persists. If it does, I’ll send the follow-up logs via email. Thanks!
Hi @lvoegtle thank you for the update. It’s good to know that this issue only occurred in one of your runs. It seems something went wrong with that particular run. I was wondering if you could try manually syncing the folder from the first run. You can do this by following the instructions [here]:
wandb sync --include-online --include-synced /path/to/wandb/run-id-timestamp
Would this work for you? also, is there any chance the images had indeed a size of 0 bytes - did you check their size in that files directory?
Hi @lvoegtle we wanted to follow up with you regarding your support request as we have not heard back from you. Please let us know if we can be of further assistance or if your issue has been resolved.
Hi @lvoegtle , since we have not heard back from you and this seems to have been resolved for follow-up runs, we are going to close this request for now. If the issue pops up again, and you would like to re-open the conversation, please let us know - and we will be more than happy to keep investigating!
