Fine tuning pretrained model in keras

Question

I want to use a pretrained imagenet VGG16 model in keras and add my own small convnet on top. I am only interested in the features, not the predictions

from keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img
from keras.applications.vgg16 import VGG16
from keras.preprocessing import image
from keras.applications.vgg16 import preprocess_input
import numpy as np
import os
from keras.models import Model
from keras.models import Sequential
from keras.layers import Convolution2D, MaxPooling2D
from keras.layers import Activation, Dropout, Flatten, Dense

load images from directory (the dir contains 4 images)

IF = '/home/ubu/files/png/'
files = os.listdir(IF)

imgs = [img_to_array(load_img(IF + p, target_size=[224,224])) for p in files]
im = np.array(imgs)

load the base model, preprocess input and get the features

base_model = VGG16(weights='imagenet', include_top=False)

x = preprocess_input(aa)
features = base_model.predict(x)

this works, and I get the features for my images on the pretrained VGG.

I now want to finetune the model and add some convolutional layers. I read https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html and https://keras.io/applications/ but cannot quite bring them together.

adding my model on top:

x = base_model.output
x = Convolution2D(32, 3, 3)(x)
x = Activation('relu')(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Convolution2D(32, 3, 3)(x)
x = Activation('relu')(x)
feat = MaxPooling2D(pool_size=(2, 2))(x)

building the complete model

model_complete = Model(input=base_model.input, output=feat)

stop base layers from being learned

for layer in base_model.layers:
layer.trainable = False

new model

model_complete.compile(optimizer='rmsprop', 
          loss='binary_crossentropy')

now fit the new model, the model is 4 images and [1,0,1,0] are the class labels. But this is obviously wrong:

model_complete.fit_generator((x, [1,0,1,0]), samples_per_epoch=100, nb_epoch=2)

ValueError: output of generator should be a tuple (x, y, sample_weight) or (x, y). Found: None

How is this done?

How would I do it if I only wanted to replace the last convolutional block (conv block5 in VGG16) instead of adding something?

How would I only train the bottleneck features?

The features output features has shape (4, 512, 7, 7). There are four images, but what is in the other dimensions? How would I reduce that to a (1,x) array?

Answer 1

Fitting model

The problem with your generator code is that the fit_generator method expects a generator function to generate the data for fitting which you don't provide. You can either define a generator as done in the tutorial that you have linked to or create the data and labels yourself and fit your model yourself:

model_complete.fit(images, labels, batch_size=100, nb_epoch=2)

where images are your generated training images and labels are the corresponding labels.

Removing last layer

Assuming you have a model variable and the "pop" method described below, you can do model = pop(model) to remove the last layer.

Training only specific layers As you have done in your code, you can do:

for layer in base_model.layers:
    layer.trainable = False

Then you can "unfreeze" and layer that you want by changing their trainable property to True.

Changing dimensions

To change the output to a 1D array you can use the Flatten layer

---

The pop method

def pop(model):
    '''Removes a layer instance on top of the layer stack.
    This code is thanks to @joelthchao https://github.com/fchollet/keras/issues/2371#issuecomment-211734276
    '''
    if not model.outputs:
        raise Exception('Sequential model cannot be popped: model is empty.')
    else:
        model.layers.pop()
        if not model.layers:
            model.outputs = []
            model.inbound_nodes = []
            model.outbound_nodes = []
        else:
            model.layers[-1].outbound_nodes = []
            model.outputs = [model.layers[-1].output]
        model.built = False

    return model