Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to set different learning rate for different layer in pytorch?

I'm doing fine-tuning with pytorch using resnet50 and want to set the learning rate of the last fully connected layer to 10^-3 while the learning rate of other layers be set to 10^-6. I know that I can just follow the method in its document:

optim.SGD([{'params': model.base.parameters()},
           {'params': model.classifier.parameters(), 'lr': 1e-3}], 
          lr=1e-2, momentum=0.9)

But is there anyway that I do not need to set the parameters layer by layer

like image 475
w.wei Avatar asked May 06 '17 08:05

w.wei


People also ask

What is the learning rate in Pytorch?

classifier 's parameters will use a learning rate of 1e-3 , and a momentum of 0.9 will be used for all parameters.

What are discriminative learning rates?

A discriminative learning rate is when you train a neural net with different learning rates for different layers.


1 Answers

You can group layers. If you want to group all linear layers, the best way to do it is use modules:

param_grp = []

for idx, m in enumerate(model.modules()):
    if isinstance(m, nn.Linear):
        param_grp.append(m.weight)
like image 137
cedrickchee Avatar answered Oct 21 '22 17:10

cedrickchee