How to use InceptionV3 bottlenecks as input in Keras 2.0

Question

I want to use bottlenecks for transfer learning using InceptionV3 in Keras. I've used some of the tips on creating, loading and using bottlenecks from https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html

My problem is that I don't know how to use a bottleneck (numpy array) as input to an InceptionV3 with a new top layer.

I get the following error:

ValueError: Error when checking input: expected input_3 to have shape (None, None, None, 3) but got array with shape (248, 8, 8, 2048)

248 refers to the total number of images in this case.

I know that this line is wrong, but I dont't know how to correct it:

model = Model(inputs=base_model.input, outputs=predictions)

What is the correct way to input the bottleneck into InceptionV3?

Creating the InceptionV3 bottlenecks:

def create_bottlenecks():
datagen = ImageDataGenerator(rescale=1. / 255)

model = InceptionV3(include_top=False, weights='imagenet')

# Generate bottlenecks for all training images
generator = datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    class_mode=None,
    shuffle=False)

nb_train_samples = len(generator.filenames)
bottlenecks_train = model.predict_generator(generator, int(math.ceil(nb_train_samples / float(batch_size))), verbose=1)
np.save(open(train_bottlenecks_file, 'w'), bottlenecks_train)

# Generate bottlenecks for all validation images
generator = datagen.flow_from_directory(
    validation_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    class_mode=None,
    shuffle=False)

nb_validation_samples = len(generator.filenames)

bottlenecks_validation = model.predict_generator(generator, int(math.ceil(nb_validation_samples / float(batch_size))), verbose=1)
np.save(open(validation_bottlenecks_file, 'w'), bottlenecks_validation)

Loading the bottlenecks:

def load_bottlenecks(src_dir, bottleneck_file):
    datagen = ImageDataGenerator(rescale=1. / 255)
    generator = datagen.flow_from_directory(
        src_dir,
        target_size=(img_width, img_height),
        batch_size=batch_size,
        class_mode='categorical',
        shuffle=False)

    num_classes = len(generator.class_indices)

    # load the bottleneck features saved earlier
    bottleneck_data = np.load(bottleneck_file)

    # get the class lebels for the training data, in the original order
    bottleneck_class_labels = generator.classes

    # convert the training labels to categorical vectors
    bottleneck_class_labels = to_categorical(bottleneck_class_labels, num_classes=num_classes)

    return bottleneck_data, bottleneck_class_labels

Starting training:

def start_training():
global nb_train_samples, nb_validation_samples

create_bottlenecks()

train_data, train_labels = load_bottlenecks(train_data_dir, train_bottlenecks_file)
validation_data, validation_labels = load_bottlenecks(validation_data_dir, validation_bottlenecks_file)

nb_train_samples = len(train_data)
nb_validation_samples = len(validation_data)

base_model = InceptionV3(weights='imagenet', include_top=False)

# add a global spatial average pooling layer
x = base_model.output
x = GlobalAveragePooling2D()(x)

# let's add a fully-connected layer
x = Dense(1024, activation='relu')(x)

# and a logistic layer -- let's say we have 2 classes
predictions = Dense(2, activation='softmax')(x)

# What is the correct input? Obviously not base_model.input.
model = Model(inputs=base_model.input, outputs=predictions)

# first: train only the top layers (which were randomly initialized)
# i.e. freeze all convolutional InceptionV3 layers
for layer in base_model.layers:
    layer.trainable = False

model.compile(optimizer=optimizers.SGD(lr=0.01, momentum=0.9), loss='categorical_crossentropy', metrics=['accuracy'])

# train the model on the new data for a few epochs
history = model.fit(train_data, train_labels,
                    epochs=epochs,
                    batch_size=batch_size,
                    validation_data=(validation_data, validation_labels),
)

Any help would be appreciated!

Answer 1

This error happens when you try to train your model with input data in a different shape from the shape your model supports.

Your model supports (None, None, None, 3), meaning:

• Any number of images
• Any height
• Any width
• 3 channels

So, you must make sure that train_data (and validation_data) matches this shape.

The system is telling that train_data.shape = (248,8,8,2048)

I see that train_data comes from load_botlenecks. Is it really supposed to be coming from there? What is train data supposed to be? An image? Something else? What is a bottleneck?

---

Your model starts in the Inception model, and the Inception model takes images.

But if bottlenecks are already results of the Inception model, and you want to feed only bottlenecks, then the Inception model should not participate of anything at all.

Start from:

inputTensor = Input((8,8,2048)) #Use (None,None,2048) if bottlenecks vary in size    
x = GlobalAveragePooling2D()(inputTensor)

.....

Create the model with:

model = Model(inputTensor, predictions)

---

The idea is:

• Inception model: Image -> Inception -> Bottlenecks
• Your model: Bottlenecks -> Model -> Labels

The combination of the two models is only necessary when you don't have the bottlenecks preloaded, but you have your own images for which you want to predict the bottlenecks first. (Of course you can work with separate models as well)

Then you're going to input only images (the bottlenecks will be created by Inception and passed to your model, everything internally):

• Combined model: Image -> Inception ->(bottlenecks)-> Model -> Labels

For that:

inputImage = Input((None,None,3))
bottleNecks = base_model(inputImage)
predictions = model(bottleNecks)

fullModel = Model(inputImage, predictions)