How to select correct value of learning rate multiplier?

Question

I want to manually select correct Learning rate in an image classification problem using Pytorch by running the model for few epochs. I have used LR scheduler to decay the learning rate and also have manipulated Learning rate in optimizer parameter group but i am unable to see any change in Loss.

Answer 1

Adjusting the learning rate and finding "the one" can be very tedious and time consuming.

Luckily for you, you are not the first one to be bothered by this issue and there are several approaches to adjust the learning rate in a more systematic way.
To name just two of these methods:

1. Hyper-gradient descent - treating the learning rate as a parameter and "learning" it as well.
2. Cyclical learning rates - searching for an "optimal" learning rate within an interval.

Answer 2

Please have a look at this. It might give you some insights. LR Finder fastai