Convolutional neural networks and 3D images

Question

I am interested in applying CNNs to 3D images (i.e. medical data). Does TensorFlow already incorporate this functionality?

Answer 1

TensorFlow now supports 3D convolution and 3D pooling in the master branch.

You can use them with 5D tensors as input with shape: [batch_size, depth, height, width, channels].

Answer 2

No, the current implementations are made for 2D images (functions like nn.conv2d). They support multiple channels (eg RGB) and it is possible to express 3D images as a multichannel 2D image (each z-slice is a channel), but this isn't always ideal. Additionally for using these sort of approaches you need substantial amounts of image data which is typically difficult to come by in the medical area.

Update: both TensorFlow and Theano (subsequently Keras, Lasagne, etc) now all support 3D operations as stated above. It is important to note that 3D operations are much more computationally and memory intensive than a similar 2D operation.

Answer 3

The TensorFlow implementation for 3D Convolutional Neural Networks has been provided with the following open source projects:

Lip Reading - Cross Audio-Visual Recognition using 3D Convolutional Neural Networks

Using 3D Convolutional Neural Networks for Speaker Verification