How to handle very large 3D data in Deep Learning

Question

I have a large input feature as 3D array of size 500x500x500 and 10000 of such samples. And the label of size 500x500x500x500. I created a model with input shape of 500x500x500 using only one Conv3D layer at input and Dense layer at output (I have my own reason for dense layer at output) , the output shape of the network is 500x500x500x500.

Below is the bare minimum model which I used:

ip = Input(shape=(500,500,500,1))  
x = Conv3D(100,3,activation="relu",padding='same')(ip)
x = Dense(500,activation="softmax")(x)
nn = Model(inputs=ip, outputs=x)

Below is the summary:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_5 (InputLayer)         (None, 500, 500, 500, 1)  0         
_________________________________________________________________
conv3d_4 (Conv3D)            (None, 500, 500, 500, 100 2800      
_________________________________________________________________
dense_4 (Dense)              (None, 500, 500, 500, 500 50500     
=================================================================
Total params: 53,300
Trainable params: 53,300
Non-trainable params: 0
_________________________________________________________________

when I run the model I got the memory error as I have 64 GB RAM and quadroP5000 nvidia GPU.

Another way to make it working was to split the input to 100s of 5x500x500 chunks thus making the network input of size 5x500x500 . Now I have 10000x100=1000000 samples of size 5x500x500. Below is the modified network:

ip = Input(shape=(5,500,500,1))  
x = Conv3D(100,3,activation="relu",padding='same')(ip)
x = Dense(500,activation="softmax")(x)
nn = Model(inputs=ip, outputs=x)

below is the summary:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_6 (InputLayer)         (None, 5, 500, 500, 1)    0         
_________________________________________________________________
conv3d_5 (Conv3D)            (None, 5, 500, 500, 100)  2800      
_________________________________________________________________
dense_5 (Dense)              (None, 5, 500, 500, 500)  50500     
=================================================================
Total params: 53,300
Trainable params: 53,300
Non-trainable params: 0
_________________________________________________________________

Clearly the total number of parameters are same, but now I am able to train the network as I am able to load the data in RAM .But the network is not able to learn as it can't see all the information at once it can see only 5 of those. The information is distributed over whole array of size 500x500x500, so network can't figure out anything looking at only one chunk of size 5x500x500. Please suggest me how to get over this. I want my network to use all the information for prediction not only one chunk.

Answer 1

You could resize your inputs before supplying them to the CNN. For example, I would go for 100x100x100 or even 50x50x50. Another option would be to add a Max/Mean/Average Pooling layer before the Convolution layer to have a sort of dimensionality reduction.

I had the same problem and I figured out that if I ran the whole process utilising only CPU it was working (but it was taking ages). I am not sure if this adds any value to your concerns.