Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to apply layer-wise learning rate in Pytorch?

Tags:

I know that it is possible to freeze single layers in a network for example to train only the last layers of a pre-trained model. What I’m looking for is a way to apply certain learning rates to different layers.

So for example a very low learning rate of 0.000001 for the first layer and then increasing the learning rate gradually for each of the following layers. So that the last layer then ends up with a learning rate of 0.01 or so.

Is this possible in pytorch? Any idea how I can archive this?

like image 691
MBT Avatar asked Aug 11 '18 16:08

MBT


People also ask

How do you find the optimal learning rate of PyTorch?

tested learning rate values (Figure 1.), you usually should find the best learning rate values somewhere around the middle of the steepest descending loss curve. In Figure 1 where loss starts decreasing significantly between LR 10−3 and 10−1, red dot indicates optimal value chosen by PyTorch Lightning.

What is the learning rate in PyTorch?

classifier 's parameters will use a learning rate of 1e-3 , and a momentum of 0.9 will be used for all parameters.

What is the discriminative learning rate?

A discriminative learning rate is when you train a neural net with different learning rates for different layers.


1 Answers

Here is the solution:

from torch.optim import Adam  model = Net()  optim = Adam(     [         {"params": model.fc.parameters(), "lr": 1e-3},         {"params": model.agroupoflayer.parameters()},         {"params": model.lastlayer.parameters(), "lr": 4e-2},     ],     lr=5e-4, ) 

Other parameters that are didn't specify in optimizer will not optimize. So you should state all layers or groups(OR the layers you want to optimize). and if you didn't specify the learning rate it will take the global learning rate(5e-4). The trick is when you create the model you should give names to the layers or you can group it.

like image 195
Salih Karagoz Avatar answered Sep 24 '22 23:09

Salih Karagoz