Menu
I was wondering how to implement biLSTM with Batch Normalization (BN) in Keras. I know that BN layer should be between linearity and nonlinearity, i.e., activation. This is easy to implement with CNN or Dense layers. But, how to do this with biLSTM?
Thanks in advance.
410
The LSTM architecture allows for models with fewer vanishing gradients and there- fore more memory than a vanilla recurrent network. Batch normalization lends a higher training speed to the model.
A new BatchNormalization layer can be added to the model after the hidden layer before the output layer. Specifically, after the activation function of the prior hidden layer.
LSTM stands for Long Short-Term Memory, a model initially proposed in 1997 [1]. LSTM is a Gated Recurrent Neural Network, and bidirectional LSTM is just an extension to that model. The key feature is that those networks can store information that can be used for future cell processing.
If you want to apply BatchNormalization over the linear outputs of an LSTM you can do it as
from keras.models import Sequential
from keras.layers.recurrent import LSTM
from keras.layers.wrappers import Bidirectional
from keras.layers.normalization import BatchNormalization
model = Sequential()
model.add(Bidirectional(LSTM(128, activation=None), input_shape=(256,10)))
model.add(BatchNormalization())
Essentially, you are removing the non-linear activations of the LSTM (but not the gate activations), and then applying BatchNormalization to the outpus.
If what you want is to apply BatchNormalization into one of the inside flows of the LSTM, such as recurrent flows, I'm afraid that feature has not been implemented in Keras.
64
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
