How to experiment with custom 2d-convolution kernels in Keras?

Question

In a default Conv2D layer with kernel_size=3 the weights of a slice of one of the filters could be named like this:

A B C
D E F
G H I

With kernel_size=5 like this:

A B C D E
F G H I J
K L M N O
P Q R S T
U V W X Y

Now I'd like to build (and train/test) a model based on conv layers with kernels like that:

A A B C C
A A B C C
D D E F F
G G H I I
G G H I I

How could the implementation of such a custom layer look like?

Answer 1

Maybe like this?

class CustomConv2D(Layer):
    def __init__(self, filters, **kwargs):
        self.filters = filters
        self.kernel_size = (3, 3)
        super(CustomConv2D, self).__init__(**kwargs)

    def build(self, input_shape):
        # only have a 3x3 kernel
        shape = self.kernel_size + (input_shape[-1], self.filters)
        self.kernel = self.add_weight(name='kernel', shape=shape,
                                      initializer='glorot_uniform')
        super(CustomConv2D, self).build(input_shape)

    def call(self, x):
        # duplicate rows 0 and 2
        dup_rows = K.stack([self.kernel[0]]*2 + [self.kernel[1]] + [self.kernel[2]]*2, axis=0)
        # duplicate cols 0 and 2
        dup_cols = K.stack([dup_rows[:,0]]*2 + [dup_rows[:,1]] + [dup_rows[:,2]]*2, axis=1)
        # having a 5x5 kernel now
        return K.conv2d(x, dup_cols)

    def compute_output_shape(self, input_shape):
        return input_shape[:-1] + (self.filters,)

The trick is to simply store only 9 weights per filter in a 3x3 kernel (hard coded, you may want to generalize it) and to duplicate the first and last rows and columns to make it a 5x5 kernel the way you want it. Then this kernel is passed to K.conv2d() just like in the original Conv2d implementation.

I tested it and it seems to be working. You may want to add other parameters like padding, bias, etc.

Answer 2

I think this does a basic version of what you intend:

from keras import backend as K

class Conv2DTiledKernel(Layer):
    def __init__(self, filters, kernel_size, multiplies, **kwargs):
        self.filters = filters
        self.kernel_size = kernel_size
        self.multiplies = multiplies
        super(Conv2DTiledKernel, self).__init__(**kwargs)
    def build(self, input_shape):
        shape = list(self.kernel_size) + [input_shape[-1], self.filters]
        self.kernel = self.add_weight(name='kernel', shape=shape,
                                      initializer='glorot_uniform')
        super(Conv2DTiledKernel, self).build(input_shape)
    def call(self, x):
        mult = list(self.multiplies) + [1, 1]
        kernel_tiled = K.tile(self.kernel, mult)
        return K.conv2d(x, kernel_tiled)
    def compute_output_shape(self, input_shape):
        return input_shape[:-1] + (self.filters,)

fitlers is the number of output channels, kernel_size the size of each kernel channel and multiplies the tiling factors. You would use it something like this:

from keras.models import Model
from keras.layers import Input, Layer

img = Input(shape=(64, 64, 3))
output = Conv2DTiledKernel(10, [1, 5], [5, 1])(img)
model = Model(inputs=img, outputs=output)

This is a pretty basic version, though. You could later add options for bias, regularizer, initalization, padding, strides, dilation, etc. You can look at the source code to see how convolutional layers are implemented in Keras. It would have been ideal if you could just subclass one of the classes so you get all the additional options for free, but I'm not sure it can be done in a practical way as the code currently stands.