I'm trying to develop an Encoder model in keras for timeseries. The shape of data is (5039, 28, 1), meaning that my seq_len is 28 and I have one feature. For the first layer of the encoder, I'm using 112 hunits, second layer will have 56 and to be able to get back to the input shape for decoder, I had to add 3rd layer with 28 hunits (this autoencoder is supposed to reconstruct its input). But I don't know what is the correct approach to connect the LSTM layers together. AFAIK, I can either add RepeatVector
or return_seq=True
. You can see both of my models in the following code. I wonder what will be the difference and which approach is the correct one?
First model using return_sequence=True
:
inputEncoder = Input(shape=(28, 1))
firstEncLayer = LSTM(112, return_sequences=True)(inputEncoder)
snd = LSTM(56, return_sequences=True)(firstEncLayer)
outEncoder = LSTM(28)(snd)
context = RepeatVector(1)(outEncoder)
context_reshaped = Reshape((28,1))(context)
encoder_model = Model(inputEncoder, outEncoder)
firstDecoder = LSTM(112, return_sequences=True)(context_reshaped)
outDecoder = LSTM(1, return_sequences=True)(firstDecoder)
autoencoder = Model(inputEncoder, outDecoder)
Second model with RepeatVector
:
inputEncoder = Input(shape=(28, 1))
firstEncLayer = LSTM(112)(inputEncoder)
firstEncLayer = RepeatVector(1)(firstEncLayer)
snd = LSTM(56)(firstEncLayer)
snd = RepeatVector(1)(snd)
outEncoder = LSTM(28)(snd)
encoder_model = Model(inputEncoder, outEncoder)
context = RepeatVector(1)(outEncoder)
context_reshaped = Reshape((28, 1))(context)
firstDecoder = LSTM(112)(context_reshaped)
firstDecoder = RepeatVector(1)(firstDecoder)
sndDecoder = LSTM(28)(firstDecoder)
outDecoder = RepeatVector(1)(sndDecoder)
outDecoder = Reshape((28, 1))(outDecoder)
autoencoder = Model(inputEncoder, outDecoder)
RepeatVector is used to repeat the input for set number, n of times. For example, if RepeatVector with argument 16 is applied to layer having input shape as (batch_size, 32), then the output shape of the layer will be (batch_size, 16, 32)
repeatVector() function is used to repeat the input n number of times in a new specified dimension. It is an inbuilt function of TensorFlow's. js library.
LSTM return_sequences=True value: The ouput is a 3D array of real numbers. The first dimension is indicating the number of samples in the batch given to the LSTM layer. The second dimension is the number of time steps in the input sequence.
LSTM autoencoder is an encoder that makes use of LSTM encoder-decoder architecture to compress data using an encoder and decode it to retain original structure using a decoder.
You will probably have to see for yourself which one is better because it depends on the problem you're solving. However, I'm giving you the difference between the two approaches.
Essentially, return_sequences=True
returns all the outputs the encoder observed in the past, while RepeatVector
repeats the very last output of the encoder.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With