Menu
Does anyone know the default activation function used in the recurrent layers in Keras? https://keras.io/layers/recurrent/
It says the default activation function is linear. But what about the default recurrent activation function. Nothing is mentioned about that. Any help would be highly appreciated. Thanks in advance
339
The ReLU function is the default activation function for hidden layers in modern MLP and CNN neural network models. We do not usually use the ReLU function in the hidden layers of RNN models. Instead, we use the sigmoid or tanh function there. We never use the ReLU function in the output layer.
relu function Applies the rectified linear unit activation function. With default values, this returns the standard ReLU activation: max(x, 0) , the element-wise maximum of 0 and the input tensor.
activation: Activation function to use. Default: hyperbolic tangent ( tanh ). If you pass None, no activation is applied (ie. "linear" activation: a(x) = x ).
The Rectified Linear Unit (ReLU) is the most commonly used activation function in deep learning. The function returns 0 if the input is negative, but for any positive input, it returns that value back.
Keras Recurrent is an abstact class for recurrent layers. In Keras 2.0 all default activations are linear for all implemented RNNs (LSTM, GRU and SimpleRNN). In previous versions you had:
linear for SimpleRNN,tanh for LSTM and GRU.
173
https://github.com/keras-team/keras/blob/master/keras/layers/recurrent.py#L2081
It mentions tanh here for version 2.3.0 :-)
36
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
