Hard-swish for TFLite

Question

I have a custom neural network written in Tensorflow.Keras and apply the hard-swish function as activation (as used in the MobileNetV3 paper):

Implementation:

def swish(x):
    return x * tf.nn.relu6(x+3) / 6

I am running quantization aware training and write a protobuf file at the end. Then, I am using this code to convert to tflite (and deploy it finally on the EdgeTPU):

tflite_convert --output_file test.tflite --graph_def_file=test.pb --inference_type=QUANTIZED_UINT8 --input_arrays=input_1 --output_arrays=conv2d_3/Sigmoid --mean_values=0 --std_dev_values=255 --default_ranges_min=0 --default_ranges_max=6

And this works perfectly, when I am not dividing by 6, however, when dividing by 6 I am getting this error:

Unimplemented: this graph contains an operator of type Div for which the quantized form is not yet implemented.

I am using TF 1.14 to train and TF 1.15 last nightly build to convert to TFLITE; I am struggling to get TF 2.x to work for some strange HDF5 incompatibilities, but it would be great if someone knows how to circumvent this issue... Thanks!

Answer 1

Since it is a constant division, you could just multiply by (a close approximation of) the inverse:

def swish(x):
    return x * tf.nn.relu6(x+3) * 0.16666667