Tensorflow: optimize over input with gradient descent

Question

I have a TensorFlow model (a convolutional neural network) which I successfully trained using gradient descent (GD) on some input data.

Now, in a second step, I would like to provide an input image as initialization then and optimize over this input image with fixed network parameters using GD. The loss function will be a different one, but this a detail.

So, my main question is how to tell the gradient descent algorithm to

• stop optimizing the network parameters
• to optimize over the input image

The first can probably done with this Holding variables constant during optimizer

Do you guys have ideas about the second point?

I guess I can recode the gradient descent algorithm myself using the TF gradient function, but my gut feeling tells me that there should be an easier way, which also allows me to benefit from more complex GD variants (Adam etc.).

Answer 1

No need for your SDG own implementation. TensorFlow provides all functions:

import tensorflow as tf
import numpy as np

# some input
data_pldhr = tf.placeholder(tf.float32)
img_op = tf.get_variable('input_image', [1, 4, 4, 1], dtype=tf.float32, trainable=True)
img_assign = img_op.assign(data_pldhr)

# your starting image
start_value = (np.ones((4, 4), dtype=np.float32) + np.eye(4))[None, :, :, None]


# override variable_getter
def nontrainable_getter(getter, *args, **kwargs):
    kwargs['trainable'] = False
    return getter(*args, **kwargs)


# all variables in this scope are not trainable
with tf.variable_scope('myscope', custom_getter=nontrainable_getter):
    x = tf.layers.dense(img_op, 10)
    y = tf.layers.dense(x, 10)

# the usual stuff
cost_op = tf.losses.mean_squared_error(x, y)
train_op = tf.train.AdamOptimizer(0.1).minimize(cost_op)

# fire up the training process
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    sess.run(img_assign, {data_pldhr: start_value})

    print(sess.run(img_op))
    for i in range(10):
        _, c = sess.run([train_op, cost_op])
        print(c)
    print(sess.run(img_op))