Restore variables that are a subset of new model in Tensorflow?

Question

I am doing an example for boosting(4 layers DNN to 5 layers DNN) via Tensorflow. I am making it with save session and restore in TF because there is a brief paragraph in TF tute: 'For example, you may have trained a neural net with 4 layers, and you now want to train a new model with 5 layers, restoring the parameters from the 4 layers of the previously trained model into the first 4 layers of the new model.', where tensorflow tute inspires in https://www.tensorflow.org/how_tos/variables/.

However, I found that nobody has asked how to use 'restore' when the checkpoint saves parameters of 4 layers but we need to put that into 5 layers, raising a red flag.

Making this in real code, I made

with tf.name_scope('fcl1'):
    hidden_1 = fully_connected_layer(inputs, train_data.inputs.shape[1], num_hidden)            
with tf.name_scope('fcl2'):
    hidden_2 = fully_connected_layer(hidden_1, num_hidden, num_hidden)                
with tf.name_scope('fclf'):
    hidden_final = fully_connected_layer(hidden_2, num_hidden, num_hidden)    
with tf.name_scope('outputl'):
    outputs = fully_connected_layer(hidden_final, num_hidden, train_data.num_classes, tf.identity)
    outputs = tf.nn.softmax(outputs)
with tf.name_scope('boosting'):
    boosts = fully_connected_layer(outputs, train_data.num_classes, train_data.num_classes, tf.identity)

where variables inside(or called from) 'fcl1' - so that I have 'fcl1/Variable' and 'fcl1/Variable_1' for weight and bias - 'fcl2', 'fclf', and 'outputl' are stored by saver.save() in the script without 'boosting' layer. However, as we now have 'boosting' layer, saver.restore(sess, "saved_models/model_list.ckpt") does not work as

NotFoundError: Key boosting/Variable_1 not found in checkpoint

I really hope to hear about this problem. Thank you. Below code is the main part of the code I am in trouble.

def fully_connected_layer(inputs, input_dim, output_dim, nonlinearity=tf.nn.relu):
    weights = tf.Variable(
        tf.truncated_normal(
            [input_dim, output_dim], stddev=2. / (input_dim + output_dim)**0.5), 
        'weights')
    biases = tf.Variable(tf.zeros([output_dim]), 'biases')
    outputs = nonlinearity(tf.matmul(inputs, weights) + biases)    

    return outputs

inputs = tf.placeholder(tf.float32, [None, train_data.inputs.shape[1]], 'inputs')
targets = tf.placeholder(tf.float32, [None, train_data.num_classes], 'targets')

with tf.name_scope('fcl1'):
    hidden_1 = fully_connected_layer(inputs, train_data.inputs.shape[1], num_hidden)            
with tf.name_scope('fcl2'):
    hidden_2 = fully_connected_layer(hidden_1, num_hidden, num_hidden)                
with tf.name_scope('fclf'):
    hidden_final = fully_connected_layer(hidden_2, num_hidden, num_hidden)    
with tf.name_scope('outputl'):
    outputs = fully_connected_layer(hidden_final, num_hidden, train_data.num_classes, tf.identity)

with tf.name_scope('error'):    
    error = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(outputs, targets))
with tf.name_scope('accuracy'):
    accuracy = tf.reduce_mean(tf.cast(
        tf.equal(tf.argmax(outputs, 1), tf.argmax(targets, 1)), 
        tf.float32))
with tf.name_scope('train'):
    train_step = tf.train.AdamOptimizer().minimize(error)

init = tf.global_variables_initializer()  
saver = tf.train.Saver()

with tf.Session() as sess:
    sess.run(init)
    saver.restore(sess, "saved_models/model.ckpt")
    print("Model restored")

    print("Optimization Starts!")
    for e in range(training_epochs):
        ...

   #Save model - save session        
    save_path = saver.save(sess, "saved_models/model.ckpt")
    ### I once saved the variables using var_list, but didn't work as well...
    print("Model saved in file: %s" % save_path)

For clarity, the checkpoint file has

fcl1/Variable:0

fcl1/Variable_1:0

fcl2/Variable:0

fcl2/Variable_1:0

fclf/Variable:0

fclf/Variable_1:0

outputl/Variable:0

outputl/Variable_1:0

As the original 4 layers model does not have 'boosting' layer.

Answer 1

It doesn't look right to read values for boosting from the checkpoint in this case and I think that's not what you want to do. Obviously you're getting error, since while restoring the variables you are first catching the list of all of the variables in your model and then you look for corresponding variables in your checkpoint, which doesn't have them.

You can restore only part of your model by defining a subset of your model variables. For example you can do it using tf.slim library. Getting the list of variables in your models:

variables = slim.get_variables_to_restore()

Now variables is a list of tensors, but for each element you can access its name attribute. Using that you can specify that you only want to restore layers other than boosting, e.g.:

variables_to_restore = [v for v in variables if v.name.split('/')[0]!='boosting'] 
model_path = 'your/model/path'

saver = tf.train.Saver(variables_to_restore)

with tf.Session() as sess:
    saver.restore(sess, model_path)

This way you will have your 4 layers restored. Theoretically you could try to catch values of one of your variables from checkpoint by creating another server that will only have boosting in variables list and renaming the chosen variable from the checkpoint, but I really don't think it's what you need here.

Since this is a custom layer for your model and you don't have this variable anywhere, just initialize it within your workflow instead of trying to import it. You can do for example by passing this argument while calling a function fully_connected:

weights_initializer = slim.variance_scaling_initializer()

You need to check details yourself though, since I'm not sure what your imports are and which function are you using here.

Generally I'd advice you to take a look at slim library, which will make it easier for you to define a model and scopes for layers (instead of defining it by with you can rather pass a scope argument while calling a function). It would look something like that with slim:

boost = slim.fully_connected(input, number_of_outputs, activation_fn=None, scope='boosting', weights_initializer=slim.variance_scaling_initializer())