Keras - How to share the convolutional part of a CNN with 2 separate FullyConnected layers

Question

I'm working on a CNN aiming at predicting 2 different volumes (systolic, and diastolic) from a set of cardiac MRI images.

First thing I have done is implementing 2 separate "sequential" CNN, each of them predicting one volume.

First defining the model :

def get_model():
    dim_img = 64
    model = Sequential()
    model.add(Activation(activation=center_normalize, input_shape=(30, dim_img, dim_img)))

    # CONVOLUTIONAL LAYERS
    model.add(Convolution2D(32, 3, 3, border_mode='same'))
    model.add(Activation('relu'))
    model.add(Convolution2D(32, 3, 3,border_mode='same'))
    model.add(Activation('relu'))
    model.add(Dropout(0.25))

    model.add(Convolution2D(64, 3, 3, border_mode='same'))
    model.add(Activation('relu'))
    model.add(Convolution2D(64, 3, 3, border_mode='same'))
    model.add(Activation('relu'))
    model.add(ZeroPadding2D(padding=(1, 1)))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
    model.add(Dropout(0.25))

    # FULLY CONNECTED LAYERS
    model.add(Flatten())
    model.add(Dense(512, W_regularizer=l2(1e-3)))
    model.add(Activation('relu'))
    model.add(Dropout(0.5))
    model.add(Dense(1))

    adam = Adam(lr=0.0001)
    model.compile(optimizer=adam, loss=root_mean_squared_error)
    return model

And then, I call two different model to predict 2 different volumes :

model_systole= get_model()
model_diastole= get_model()
print('Fitting systole model...')
hist_systole = model_systole.fit_generator(generator_train_sys, nb_epoch = 1,samples_per_epoch=N_train,validation_data=generator_validate_sys,nb_val_samples=N_validate)

print('Fitting Diastole model...')
hist_diastole = model_diastole.fit_generator(generator_train_dia,nb_epoch = 1,samples_per_epoch=N_train,validation_data=generator_validate_dia, nb_val_samples=N_validate)

However, I thus have to train two different models, with 2 convolutional parts.

===> I would like to share the convolutional part of my networks, and add to it 2 different FC layers to predict both my volumes (1 unique convolutional part and two separate FC layers sharing the convolutional part)

Would you know how to do this with Keras ? Do I have to switch to Keras Graph mode ?

Thank you very much for your help.

Answer 1

When you call get_model() twice, it creates two different instances. That means they will be initialized with two different initial set of weights. Also, they are different models, so they are in no way related during back-propagation.

You will have to switch to the Model class. I will strongly suggest you to update Keras so that you get the Model class instead of Graph.

input1 = Input(30 ,64, 64)
input2 = Input(30 ,64, 64)
conv1 = Conv2D(32, (3,3), padding='same', activation='relu') # shared

model1 = conv1(input1)
model2 = conv1(input2)
model1 = Flatten()(model1)
model2 = Flatten()(model2)

# following dense layers do not share the weight
model1 = Dense(512, activation='relu', kernel_regularizer=l2(1e-3))(model1)
model2 = Dense(512, activation='relu', kernel_regularizer=l2(1e-3))(model2)
model1 = Dense(1)(model1)
model2 = Dense(1)(model2)

model = Model(inputs=[input1, input2], outputs=[model1, model2])
model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])

Basically, layers are now callable. So I first created one convolutional layer and called it with two different inputs, so its weights are shared. However, I called a two dense layers by instantiating them twice (like Dense(512)), so they are not shared.