Keras - use part of the input at later stage in sequential model

Question

I'm training a CNN. My input is an image and a few metadata. I'd like to train a CNN that only looks at the image in the convolutional stages, and then uses the output of the convolutional stages and the metadata in the final dense layers.

metadata -----------------
                          |-> dense -> output
image    -> Convolutions -

How can I do this with Keras? Can I feed input that has not a rectangular shape?

For instance if the image is (255, 255, 3) and the metadata (10) how would this work?

I've found this issue that seems related but I don't get how they split the input and merge the second part with the intermediate output later on.

Answer 1

You need to use the Functional API with a multi-input model.

An example could be:

from keras.layers import Input, Conv1D, Dense, concatenate
#Image data
conv_input = Input(shape=conv_input_shape)
conv_output = Conv1D(nfilters,kernel_shape)(conv_input)

#Metadata
metadata_input = Input(shape=metadata_shape)

#Merge and add dense layer
merge_layer = concatenate([metadata_input, conv_output])
main_output = Dense(dense_size)(merge_layer)

# Define model with two inputs
model = Model(inputs=[conv_input, metadata_input], outputs=[main_output])

Hope this helps!