Menu
One can apply recurrent dropout onto basic LSTM or GRU layers in Keras by passing its value as a parameter of the layer.
CuDNNLSTM and CuDNNGRU are LSTM and GRU layers that are compatible with CUDA. The main advantage is that they are 10 times faster during training. However they lack some of the beauty of the LSTM or GRU layers in Keras, namely the possibility to pass dropout or recurrent dropout values.
While we can add Dropout layers directly in the model, it seems we cannot do that with Recurrent Dropout.
My question is then the following : How to add recurrent dropout to CuDNNGRU or CuDNNLSTM in Keras ?
684
One can apply recurrent dropout onto basic LSTM or GRU layers in Keras by passing its value as a parameter of the layer. CuDNNLSTM and CuDNNGRU are LSTM and GRU layers that are compatible with CUDA. The main advantage is that they are 10 times faster during training.
Recurrent Dropout is a regularization method for recurrent neural networks. Dropout is applied to the updates to LSTM memory cells (or GRU states), i.e. it drops out the input/update gate in LSTM/GRU.
So, you use Dropout the same way you would use in any fully connected network. It drops a different group of features for each sample. A dropout as an argument to the LSTM has a lot of differences. It generates 4 different dropout masks, for creating different inputs for each of the different gates.
Long Short-Term Memory Network or LSTM, is a variation of a recurrent neural network (RNN) that is quite effective in predicting the long sequences of data like sentences and stock prices over a period of time. It differs from a normal feedforward network because there is a feedback loop in its architecture.
I don't think we can have it as it is not even supported in the low level (i.e. cuDNN). From François Chollet creator of Keras:
Recurrent dropout is not implemented in cuDNN RNN ops. At the cuDNN level. So we can't have it in Keras.
The dropout option in the cuDNN API is not recurrent dropout (unlike what is in Keras), so it is basically useless (regular dropout doesn't work with RNNs).
Actually using such dropout in a stacked RNN will wreck training.
169
You can use kernel_regularizer and recurrent_regularizer for prevent overfitting, i am using L2 regularizers and i am having good results.
44
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
