How to modify the Imagenet Caffe Model?

Question

I would like to modify the ImageNet caffe model as described bellow:

As the input channel number for temporal nets is different from that of spatial nets (20 vs. 3), we average the ImageNet model filters of first layer across the channel, and then copy the average results 20 times as the initialization of temporal nets.

My question is how can I achive the above results? How can I open the caffe model to be able to do those changes to it?

I read the net surgery tutorial but it doesn't cover the procedure needed.

Thank you for your assistance!

AMayer

Answer 1

The Net Surgery tutorial should give you the basics you need to cover this. But let me explain the steps you need to do in more detail:

1. Prepare the .prototxt network architectures: You need two files: the existing ImageNet .prototxt file, and your new temporal network architecture. You should make all layers except the first convolutional layers identical in both networks, including the names of the layers. That way, you can use the ImageNet .caffemodel file to initialize the weights automatically.

   As the first conv layer has a different size, you have to give it a different name in your .prototxt file than it has in the ImageNet file. Otherwise, Caffe will try to initialize this layer with the existing weights too, which will fail as they have different shapes. (This is what happens in the edit to your question.) Just name it e.g. conv1b and change all references to that layer accordingly.

2. Load the ImageNet network for testing, so you can extract the parameters from the model file:

   net = caffe.Net('imagenet.prototxt', 'imagenet.caffemodel', caffe.TEST)
   

3. Extract the weights from this loaded model.

   conv_1_weights = old_net.params['conv1'][0].data
   conv_1_biases = old_net.params['conv1'][1].data
   

4. Average the weights across the channels:

   conv_av_weights = np.mean(conv_1_weights, axis=1, keepdims=True)
   

5. Load your new network together with the old .caffemodel file, as all layers except for the first layer directly use the weights from ImageNet:

   new_net = caffe.Net('new_network.prototxt', 'imagenet.caffemodel', caffe.TEST)
   

6. Assign your calculated average weights to the new network

   new_net.params['conv1b'][0].data[...] = conv_av_weights
   new_net.params['conv1b'][1].data[...] = conv_1_biases
   

7. Save your weights to a new .caffemodel file:

   new_net.save('new_weights.caffemodel')