How to implement PReLU activation in Tensorflow?

Question

The Parametric Rectified Linear Unit (PReLU) is an interesting and widely used activation function. It seems that Tensorflow (reference link) does not provide PReLU. I know that the higher level libraries, such as Keras and TFLearn, has the implementation of it.

I would like to know how to implement PReLU in Tensorflow?

Answer 1

The implementation of PReLU seems straight-forward based on the PreLU implementations (see: Keras, TFLearn and TensorLayer) of the higher level libraries. My code is as follows:

def parametric_relu(_x):
  alphas = tf.get_variable('alpha', _x.get_shape()[-1],
                       initializer=tf.constant_initializer(0.0),
                        dtype=tf.float32)
  pos = tf.nn.relu(_x)
  neg = alphas * (_x - abs(_x)) * 0.5

  return pos + neg