why is SeparableConv2D slower than Conv2D?

Question

I've been experimenting with different types of convolution layers, to check their computational speeds. my code, in the beginning, was as follows.

def conv_block_A(layer):
    block = tf.keras.layers.Conv2D(filters=128, kernel_size=3, strides=1, padding='same')(layer)
    block = tf.keras.layers.Conv2D(filters=196, kernel_size=3, strides=1, padding='same')(block)
    block = tf.keras.layers.Conv2D(filters=128, kernel_size=3, strides=1, padding='same')(block)
    block = tf.keras.layers.BatchNormalization(momentum=0.8)(block)
    block = tf.keras.layers.LeakyReLU(alpha=0.2)(block)

    return block

after going through a few blogs, I changed my code as

def conv_block_A(layer):
    block = tf.keras.layers.SeparableConv2D(filters=128, kernel_size=3, strides=1, padding='same')(layer)
    block = tf.keras.layers.SeparableConv2D(filters=196, kernel_size=3, strides=1, padding='same')(block)
    block = tf.keras.layers.SeparableConv2D(filters=128, kernel_size=3, strides=1, padding='same')(block)
    block = tf.keras.layers.BatchNormalization(momentum=0.8)(block)
    block = tf.keras.layers.LeakyReLU(alpha=0.2)(block)

    return block

the training process became twice as fast on CPUs, but, the training has become very slow on Tesla T4. what could be the reason?

Answer 1

It's a known issue with GPU's it was fixed in #33836. Also, you should update your GPU driver. As a rule of thumbs speedup by doing separable convolution is more noticeable with large kernel sizes because of the overhead involving doing two convolutions might be larger than the speedups.