extracting Bottleneck features using pretrained Inceptionv3 - differences between Keras' implementation and Native Tensorflow implementation

Question

(Apologies for the long post)

All,

I want to use the bottleneck features from a pretrained Inceptionv3 model to predict classification for my input images. Before training a model and predicting classification, I tried 3 different approaches for extracting the bottleneck features.

My 3 approaches yielded different bottleneck features (not just in values but even the size was different).

1. Size of my bottleneck features from Approach 1 and 2: (number of input images) x 3 x 3 x 2048

   Size of my bottleneck features from Approach 3: (number of input images) x 2048

   Why are the sizes different between the Keras based Inceptionv3 model and the native Tensorflow model? My guess is that when I say include_top=False in Keras, I'm not extracting the 'pool_3/_reshape:0' layer. Is this correct? If yes, how do I extract the 'pool_3/_reshape:0' layer in Keras? If my guess is incorrect, what 'am I missing?

2. I compared the bottleneck feature values from Approach 1 and 2 and they were significantly different. I think I'm feeding it the same input images because I resize and rescale my images before I even read it as input for my script. I have no options for my ImageDataGenerator in Approach 1 and according to the documentation for that function all the default values do not change my input image. I have set shuffle to false so I assumed that predict_generator and predict are reading images in the same order. What 'am I missing?

---

Please note:

My inputs images are in RGB format (so number of channels = 3) and I resized all of them to 150x150. I used the preprocess_input function in inceptionv3.py to preprocess all my images.

def preprocess_input(image):
    image /= 255.
    image -= 0.5
    image *= 2.
    return image

---

Approach 1: Used Keras with tensorflow as backend, an ImageDataGenerator to read my data and model.predict_generator to compute bottleneck features

I followed the example (Section Using the bottleneck features of a pre-trained network: 90% accuracy in a minute) from Keras' blog. Instead of VGG model listed there I used Inceptionv3. Below is the snippet of code I used

(code not shown here but what i did before the code below) : read all input images, resize to 150x150x3, rescale according to the preprocessing_input function mentioned above, save the resized and rescaled images

train_datagen = ImageDataGenerator() 
train_generator = train_datagen.flow_from_directory(my_input_dir, target_size=(150,150),shuffle=False, batch_size=16)

# get bottleneck features
# use pre-trained model and exclude top layer - which is used for classification
pretrained_model = InceptionV3(include_top=False, weights='imagenet', input_shape=(150,150,3))
bottleneck_features_train_v1 = pretrained_model.predict_generator(train_generator,len(train_generator.filenames)//16)

Approach 2: Used Keras with tensorflow as backend, my own reader and model.predict to compute bottleneck features

Only difference between this approach and earlier one is that I used my own reader to read the input images. (code not shown here but what i did before the code below) : read all input images, resize to 150x150x3, rescale according to the preprocessing_input function mentioned above, save the resized and rescaled images

# inputImages is a numpy array of size <number of input images x 150 x 150 x 3>
inputImages = readAllJPEGsInFolderAndMergeAsRGB(my_input_dir)

# get bottleneck features
# use pre-trained model and exclude top layer - which is used for classification
pretrained_model = InceptionV3(include_top=False, weights='imagenet', input_shape=(img_width, img_height, 3))
bottleneck_features_train_v2 = pretrained_model.predict(trainData.images,batch_size=16)

Approach 3: Used tensorflow (NO KERAS) compute bottleneck features

I followed retrain.py to extract bottleneck features for my input images. Please note that that the weights from that script can be obtained from (http://download.tensorflow.org/models/image/imagenet/inception-2015-12-05.tgz)

As mentioned in that example, I used the bottleneck_tensor_name = 'pool_3/_reshape:0' as the layer to extract and compute bottleneck features. Similar to the first 2 approaches, I used resized and rescaled images as input to the script and I called this feature list bottleneck_features_train_v3

Thank you so much

Answer 1

Different results between 1 and 2

Since you haven't shown your code, I (maybe wrongly) suggest that the problem is that you might not have used preprocess_input when declaring ImageDataGenerator ?

from keras.applications.inception_v3 import preprocess_input

train_datagen = ImageDataGenerator(preprocessing_function=preprocess_input) 

Make sure, though, that your saved image files range from 0 to 255. (Bit depth 24).

Different shapes between 1 and 3

There are three possible types of model in this case:

• include_top = True -> this will return classes
• include_top = False (only) -> this implies in pooling = None (no final pooling layer)
• include_top = False, pooling='avg' or ='max' -> has a pooling layer

So, your declared model without an explicit pooling=something doesn't have the final pooling layer in keras. Then the outputs will still have the spatial dimensions.

Solve that simply by adding a pooling at the end. One of these:

pretrained_model = InceptionV3(include_top=False, pooling = 'avg', weights='imagenet', input_shape=(img_width, img_height, 3))
pretrained_model = InceptionV3(include_top=False, pooling = 'max', weights='imagenet', input_shape=(img_width, img_height, 3))

Not sure which one the model in the tgz file is using.

As an alternative, you can also get another layer from the Tensorflow model, the one coming immediately before 'pool_3'.

Answer 2

You can look into the Keras implementation of inceptionv3 here: https://github.com/keras-team/keras/blob/master/keras/applications/inception_v3.py

so, the default parameter is:

def InceptionV3(include_top=True,
                weights='imagenet',
                input_tensor=None,
                input_shape=None,
                pooling=None,
                classes=1000):

Notice that default for pooling=None, then when building the model, the code is:

if include_top:
    # Classification block
    x = GlobalAveragePooling2D(name='avg_pool')(x)
    x = Dense(classes, activation='softmax', name='predictions')(x)
else:
    if pooling == 'avg':
        x = GlobalAveragePooling2D()(x)
    elif pooling == 'max':
        x = GlobalMaxPooling2D()(x)

# Ensure that the model takes into account
# any potential predecessors of `input_tensor`.
if input_tensor is not None:
    inputs = get_source_inputs(input_tensor)
else:
    inputs = img_input
# Create model.
model = Model(inputs, x, name='inception_v3')

So if you do not specify the pooling the bottleneck feature is extracted without any pooling, you need to specify if you want to get an average pooling or max pooling on top of these feature.