Hey good people. Here is a post about tips and tweaks you can employ to make transfer Learning work just right.
Most blog posts I have read about this topic just suggest changing the last linear layer and freezing all other parameters but there is more tweaks you can try
E.g
Discriminative Learning Rates, Don’t Freeze Batch Norm layers, Unfreezing the model after a few epochs, using a custom and better head (classifier)
Unfreezing just batchnorm makes a ton of sense – we usually expect the distribution in the domain to shift when we move to a new task (especially when it’s a much narrower task, rather than just a different task), and being able to capture at least the changes in mean and std seems like it’d help a ton without incurring too much additional cost.
Thanks for sharing!
Yes I’m surprised most tutorials suggest the opposite, that we should keep the BatchNorm frozen so as not to hurt the previous running statistics