Keras' convolution layer on images coming from circular/cyclic domain [closed]

Question

The Need

Hello, I am experimenting the usage of CNNs on images which come from a cylindric domain, so I am interested to apply the Convolution layer in a circular (or cyclic) way. I mean a convolution layer that instead of padding the image with zeros would just wrap around the image (or the kernel around the image).

Thoughts, Searches and Ideas

Coming from a signal processing background I expected it were already covered: in fact when calculations are performed in the frequency domain (i.e. by means of DFT) this circularity comes from free, and rather extra effort (there called "zero padding") is required to avoid circular equivocation.

Well, ok: I learned that, because of kernel size being normally quite small, calculations can be done more conveniently in the base domain. I can't figure out any good reason why this should be unfeasible, so I hoped that some direct way to do a cyclic convolution existed.

But I found no coverage for this function, neither in Keras nor in TensorFlow docs. Moreover, I found little or no really relevant discussion about this around: Torch7 discussion

So I am left with the following options:

• implement my own CyclicConv2D layer by subclassing Keras' layer.Layer class, as described here
• submit my own pull request for an experimental new feature, as described here. Shortly, I would think of adding some "wrap" value for the "padding=" parameter, or adding a new "wrap=" parameter. It should specify along which axes the image should wrap around. This in fact would likely be needed only along one dimension, the circular one, not both.

The Questions

Is there any more straightforward option, or source of information that I should address first?

Otherwise, where can I find any advice how to implement the former?

For the latter I must admit that I described it more to stimulate some collective thinking how it should be implemented from a user standpoint, but I don't really feel able to contribute a good pull request (i.e. one including good code). Anyway I will appreciate any advice where to start from.

Answer 1

I'm implementing something like this so though I'd add code. I think the simplest way to actually implement the wrapped padding is to use Numpy pad function with the "wrap" option. For example with

input = np.array([[1,2,3],[4,5,6],[7,8,9]])
kernel = [1,1]
#We want symmetrical padding (same top and bottom)
# and np.pad format ((before_1, after_1), … (before_N, after_N))
pad = [[i,i] for i in kernel]
padded_input = np.pad(input, pad, "wrap")

which gives,

array([[9, 7, 8, 9, 7],
       [3, 1, 2, 3, 1],
       [6, 4, 5, 6, 4],
       [9, 7, 8, 9, 7],
       [3, 1, 2, 3, 1]])

It looks like creating a custom layer similar to ZeroPadding2D called something like CyclicPadding2D may then be the best idea to minimise changes to Keras code, like so,

kernel = [7,7]
model = Sequential()
model.add(CyclicPadding2D(kernel, input_shape=(224, 224, 3)))
model.add(Conv2D(32, kernel=kernel, padding="valid"))
model.build()

You can also use this between both pooling and conv layers. The code in CyclicPadding2D would probably need to consider input format (channels, batch, etc) with something like,

if self.data_format is "channels_last":
    #(batch, depth, rows, cols, channels)
    pad = [[0,0]] + [[i,i] for i in self.kernel] + [[0,0]]
elif self.data_format is "channels_first":
    #(batch, channels, depth, rows, cols)
    pad = [[0, 0], [0, 0]] + [[i,i] for i in self.kernel]
inputs = np.pad(inputs,  pad, "wrap")

This is similar to what the Keras Numpy backend does with option "constant" hardwired while the tensorflow backend supplied no option and so defaults to constant (although interestingly tf.pad provides a reflect option).

Looking at the Keras source, perhaps something like this could be added as a feature, by simply putting the code above in the call function of the _conv when a padding option is something like "periodic". That said, simply adding a new padding layer is probably the most flexible solution.