There should be a way to attach variables to the logger, that you can modify live from from controls in the dashboard.
When you have a running job and are monitoring the progress, you sometimes want to adjust the learning rate or other hyperparameter (should we switch to fine-tuning mode, etc.).
This is a bit of a unspoken black-magic deep learning technique. However, if you read papers from Meta, etc. or talk to hardcore old-school practitioners, they have these super long-running difficult optimization problems, and say something like: “Well we trained the generator for X thousand epochs, then we enabled the discriminator, then Y thousand epochs later we dropped the learning rate, etc.” This is ideally done by monitoring a live, running job and modifying the variables in situ.
- The non-agile way to do this is let your run go for a while, decide afterwards that you should have changed something at some point in time, code that, run it again and cross your fingers. This is obviously pretty slow and requires luck.
- A hacky way to do this is to create a DSL with sentinel files that the running job reads and applies. However, the workflow is useful enough that there should be a common way to do this.
I’m not aware of any logging library that does this. So it would make great blog posts to show off and attract more users.