What is the best way of combining clinical and Image data in a Deep Learning approach?

Question

I have CT images from patients and applied a CNN to those images to predict diseases. I would like to combine my clinical data with my image data in a CNN approach, is that possible? My clinical data has information like age, sex, dates, smoker, all of them are numbers, like 1 for smoker and 0 not.

Answer 1

Have a look at, for example, this paper where they combine features from a CNN with text data. In that paper, the CNN is already pre-trained (i.e., the CNN is essentially a featurizer), but you could clearly learn all in one go. The idea would always be to

• Run the image in your input through the convolution/subsampling layers
• Just before your final fully connected (decision) layer, concatenate the other features you have available
• Feed all (pre-processed image and other features) into the decision layer.

So the answer is "yes, certainly", the details depend on which framework you are using.

Answer 2

As far as I know CNN is extremely suitable for image data, but not for other data.

A solution to your problem would be to 'color' your images with the clinical data. (In image recognition CNNs, usually an input image is split into 3 color layers: red, grey and blue. See: http://cs231n.github.io/convolutional-networks/)

Let's say your input data is a 32x32 pixel 8-bit greyscale image (so 1 color layer). I propose to add each clinical data variable as a 'color' layer. All input values in the same color layer should be the same.

Whether each layer should be the same size as the image, or if you can get away with a single pixel, I'm not sure, but at least you can treat the clinical data as an 'image' alongside the CT images.