In TensorFlow, is it possible to use different learning rate for different part of the network?

Question

The use case I have in mind is to add more layers to a pre-trained network, and I would like to tune the entire net. However, I'd like the newly added layers to have a bigger learning rate than the existing one. Is it possible to do this in TensorFlow?

Answer 1

You could use a similar approach mentioned here

Basically set a different var scope around each of the parts of the network you want to train with a separate learning rate then:

optimizer1 = tf.train.AdagradOptimzer(0.0001)
optimizer2 = tf.train.AdagradOptimzer(0.01)

first_train_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES,
                                 "scope/prefix/for/first/vars")
first_train_op = optimizer1.minimize(cost, var_list=first_train_vars)

second_train_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES,
                                  "scope/prefix/for/second/vars")                     
second_train_op = optimizer2.minimize(cost, var_list=second_train_vars)