Convolutional Neural Network (CNN) with max-pooling

Question

I've implemented a simple CNN program with Python that can machine learn on the MNIST data set. I've implemented 3 layers:

1. ConvPoolLayer, which convolves and then does mean pooling
2. FullyConnectedLayer, which is a fully connected hidden layer
3. SoftmaxLayer, which basically gives the softmax output of the network

It's in the ConvPoolLayer that I've implemented mean pooling. Here's the line of code that does mean pooling during forward propagation:

# 'activation' is a numpy array of 3D activations from the convolutional code (not shown here)     
skimage.measure.block_reduce(activation, block_size=(1, 1, 2, 2), func=np.mean) 

And here's the equivalent back-propagation code:

# delta is a numpy array of 3D error matrices back-propagated from the upper layers
delta = delta.repeat(2, axis=2).repeat(2, axis=3)

All it's doing is just upscaling the error.

My question is, how do I implement the backpropagation for max pooling without loss in performance? Or, is there a better way to do this without a function call? I get around ~90-95% accuracy after a few iterations with mean pooling, so I'd like to see how max pooling affects performance.

If there are any NumPy tricks that can be applied here, I would be glad to learn them. I want to understand myself what happens in a CNN, why things work the way they do, and whether operations can be optimised, so using frameworks isn't an option for me.

Thanks for the help!

Answer 1

1. [updated] For forward propagation for max pooling use :

   skimage.measure.block_reduce(activation, block_size=(1, 1, 2, 2), func=np.max)

2. Your backpropagation for mean pooling is not fully correct. You should devide delta by number of pooled cells (4 in your case). See equation on slide 11 at http://www.slideshare.net/kuwajima/cnnbp

3. To propagate max pooling you need to assign delta only to cell with highest value in forward pass. Hence, during the forward pass of a pooling layer it is common to keep track of the index of the max activation (sometimes also called the switches) so that gradient routing is efficient during backpropagation. See http://cs231n.github.io/convolutional-networks/#pool

Very inefficient way to implement this:

#forward
activationPrevious = np.copy(activation)
skimage.measure.block_reduce(activation, block_size=(1, 1, 2, 2), func=np.max)
maxs = activations.repeat(2, axis=2).repeat(2, axis=3)
mask = np.equal(activationPrevious, maxs).astype(int)

#backward
delta = delta.repeat(2, axis=2).repeat(2, axis=3)
delta = np.multiply(delta, mask)