Integrating Keras model into TensorFlow

Question

I am trying to use a pre-trained Keras model within TensorFlow code, as described in this Keras blog post under section II: Using Keras models with TensorFlow.

I want to use the pre-trained VGG16 network available in Keras to extract convolutional feature maps from images, and add my own TensorFlow code over that. So I've done this:

import tensorflow as tf
from tensorflow.python.keras.applications.vgg16 import VGG16, preprocess_input
from tensorflow.python.keras import backend as K

# images = a NumPy array containing 8 images

model = VGG16(include_top=False, weights='imagenet')
inputs = tf.placeholder(shape=images.shape, dtype=tf.float32)
inputs = preprocess_input(inputs)
features = model(inputs)

with tf.Session() as sess:
    K.set_session(sess)
    output = sess.run(features, feed_dict={inputs: images})
    print(output.shape)

However, this gives me an error:

FailedPreconditionError: Attempting to use uninitialized value block1_conv1_2/kernel
     [[Node: block1_conv1_2/kernel/read = Identity[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](block1_conv1_2/kernel)]]
     [[Node: vgg16_1/block5_pool/MaxPool/_3 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_132_vgg16_1/block5_pool/MaxPool", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Instead, if I run an initializer op before running the network:

with tf.Session() as sess:
    K.set_session(sess)
    tf.global_variables_initializer().run()
    output = sess.run(features, feed_dict={inputs: images})
    print(output.shape)

Then I get the expected output:

(8, 11, 38, 512)

My question is, upon running tf.global_variables_initializer(), have the variables been initialized randomly or with the ImageNet weights? I ask this because the blog post referenced above does not mention that an initializer needs to be run when using pre-trained Keras models, and indeed it makes me feel a bit uneasy.

I suspect that it does use the ImageNet weights, and that one needs to run the initializer only because TensorFlow requires all variables to be explicitly initialized. But this is just a guess.

Answer 1

TLDR

When using Keras,

1. Avoid using Session if you can (in the spirit of agnostic Keras)
2. Use Keras-handled Session through tf.keras.backend.get_session otherwise.
3. Use Keras' set_session for advanced uses (e.g. when you need profiling or device placement) and very early in your program — contrary to common practice and good usage in "pure" Tensorflow.

More about that

Variables must be initialized before they can be used. Actually, it's a bit more subtle than that: Variables must be initialized in the session they are used. Let's look at this example:

import tensorflow as tf

x = tf.Variable(0.)

with tf.Session() as sess:
    tf.global_variables_initializer().run()
    # x is initialized -- no issue here
    x.eval()
    
with tf.Session() as sess:
    x.eval()
    # Error -- x was never initialized in this session, even though
    # it has been initialized before in another session

So it shouldn't come as a surprise that variables from your model are not initialized, because you create your model before sess.

However, VGG16 not only creates initializer operations for the model variables (the ones you are calling with tf.global_variables_initializer), but actually does call them. Question is, within which Session?

Well, since none existed at the time you built your model, Keras created a default one for you, that you can recover using tf.keras.backend.get_session(). Using this session now works as expected because variables are initialized in this session:

with tf.keras.backend.get_session() as sess:
    K.set_session(sess)
    output = sess.run(features, feed_dict={inputs: images})
    print(output.shape)

Note that you could also create your own Session and provide it to Keras, through keras.backend.set_session — and this is exactly what you have done. But, as this example shows, Keras and TensorFlow have different mindsets.

A TensorFlow user would typically first construct a graph, then instantiate a Session, perhaps after freezing the graph.

Keras is framework-agnostic and does not have this built-in distinction between construction phases — in particular, we learned here that Keras may very well instantiate a Session during graph construction.

For this reason, when using Keras, I would advise against managing a tf.Session yourself and instead rely on tf.keras.backend.get_session if you need to handle TensorFlow specific code that requires a tf.Session.