Force symmetry for a TensorFlow conv2d kernel

Question

I'd like to enforce symmetry in the weights within a Variable. I really want an approximate circular symmetry. However, I could imagine either row or column enforced symmetry.

The goal is to reduce training time by reducing the number of free variables. I know my problem would like a symmetric array but I might want to include both symmetric and "free" variables. I am using conv2d now, so I believe I need to keep using it.

Answer 1

Here is a function that creates a kernel symmetric with respect to reflection over its center row:

def SymmetricKernels(height,width,in_channels,out_channels,name=None):
    half_kernels = tf.Variable(initial_value=tf.random_normal([(height+1)//2,width,in_channels,out_channels]))
    half_kernels_reversed = tf.reverse(half_kernels[:(height//2),:,:,:],[0])
    kernels = tf.concat([half_kernels,half_kernels_reversed],axis=0,name=name)
    return kernels

Usage example:

w = SymmetricKernels(5,5,1,1)
sess = tf.InteractiveSession()
sess.run(tf.global_variables_initializer())
w_ = sess.run(w)
w_[:,:,0,0]
# output:
# [[-1.299 -1.835 -1.188  0.093 -1.736]
#  [-1.426 -2.087  0.434  0.223 -0.65 ]
#  [-0.217 -0.802 -0.892 -0.229  1.383]
#  [-1.426 -2.087  0.434  0.223 -0.65 ]
#  [-1.299 -1.835 -1.188  0.093 -1.736]]

The idea is to use tf.Variable() to create only the upper half variables of the kernels (half_kernels), and then form the symmetric kernels as a concatenation of the upper half and its reflected version.

This idea can be extended to create also kernels with both left-right and up-down symmetries.

Answer 2

Another thing you can try is to tie the net's hands by convolving twice, reusing the kernel but flipping it for the second convolution (untested code):

def symmetric_convolution(input_tensor, n_filters, size, name, dilations=[1,1,1,1]):
    with tf.variable_scope("", reuse=tf.AUTO_REUSE):
        kernel = tf.get_variable(shape=[*size, input_tensor.shape[-1], n_filters], name='conv_kernel_' + name, ...)
        lr_flipped_kernel = tf.reverse(kernel, axis=[1], name='conv_kernel_flipped_lr_' + name)

    conv_l = tf.nn.conv2d(input=input_tensor, filter=kernel, strides=[1, 1, 1, 1], padding='SAME', dilations=dilations)
    conv_r = tf.nn.conv2d(input=input_tensor, filter=lr_flipped_kernel, strides=[1, 1, 1, 1], padding='SAME', dilations=dilations)

    return tf.reduce_max(tf.concat([conv_l, conv_r], axis=-1), keepdims=True, axis=[-1])

You can add in biases, activations, etc. as needed. I've used something similar in the past – reduce_max will allow your kernel to take whatever shape, and effectively give you two convolutions for one; if you use reduce_sum instead, any asymmetries will average out quite quickly and your kernel will be symmetric. What works best will depend on your use case.