Hi team, I would like to bring you attention that log output is missing during my recent training.
Could you help me investigate this since I do not change other system set up. Logger should be enabled.
Thanks
Normal log out put
Log missing
Hello @yanzijiang97 !
I looked into your runs and was curious by the 42s runtime of the run with the missing logs. Was this run ever resumed? If so, the resumed runs will overwrite part of the logs with the new resumed runs logs.
Also, if you could provide a code snippet for reproduction, I would gladly look at the code and reproduce it on my end.