Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to connect LSTM layers in Keras, RepeatVector or return_sequence=True?

I'm trying to develop an Encoder model in keras for timeseries. The shape of data is (5039, 28, 1), meaning that my seq_len is 28 and I have one feature. For the first layer of the encoder, I'm using 112 hunits, second layer will have 56 and to be able to get back to the input shape for decoder, I had to add 3rd layer with 28 hunits (this autoencoder is supposed to reconstruct its input). But I don't know what is the correct approach to connect the LSTM layers together. AFAIK, I can either add RepeatVector or return_seq=True. You can see both of my models in the following code. I wonder what will be the difference and which approach is the correct one?

First model using return_sequence=True:

inputEncoder = Input(shape=(28, 1))
firstEncLayer = LSTM(112, return_sequences=True)(inputEncoder)
snd = LSTM(56, return_sequences=True)(firstEncLayer)
outEncoder = LSTM(28)(snd)

context = RepeatVector(1)(outEncoder)
context_reshaped = Reshape((28,1))(context)

encoder_model = Model(inputEncoder, outEncoder)
firstDecoder = LSTM(112, return_sequences=True)(context_reshaped)
outDecoder = LSTM(1, return_sequences=True)(firstDecoder)

autoencoder = Model(inputEncoder, outDecoder)

Second model with RepeatVector:

inputEncoder = Input(shape=(28, 1))
firstEncLayer = LSTM(112)(inputEncoder)
firstEncLayer = RepeatVector(1)(firstEncLayer)
snd = LSTM(56)(firstEncLayer)
snd = RepeatVector(1)(snd)
outEncoder = LSTM(28)(snd)
encoder_model = Model(inputEncoder, outEncoder)

context = RepeatVector(1)(outEncoder)
context_reshaped = Reshape((28, 1))(context)

firstDecoder = LSTM(112)(context_reshaped)
firstDecoder = RepeatVector(1)(firstDecoder)
sndDecoder = LSTM(28)(firstDecoder)

outDecoder = RepeatVector(1)(sndDecoder)
outDecoder = Reshape((28, 1))(outDecoder)

autoencoder = Model(inputEncoder, outDecoder)
like image 637
Birish Avatar asked Aug 08 '18 14:08

Birish


People also ask

What is RepeatVector in Lstm?

RepeatVector is used to repeat the input for set number, n of times. For example, if RepeatVector with argument 16 is applied to layer having input shape as (batch_size, 32), then the output shape of the layer will be (batch_size, 16, 32)

What does repeat vector do?

repeatVector() function is used to repeat the input n number of times in a new specified dimension. It is an inbuilt function of TensorFlow's. js library.

What is Return_sequences in Lstm?

LSTM return_sequences=True value: The ouput is a 3D array of real numbers. The first dimension is indicating the number of samples in the batch given to the LSTM layer. The second dimension is the number of time steps in the input sequence.

What is Lstm Autoencoder?

LSTM autoencoder is an encoder that makes use of LSTM encoder-decoder architecture to compress data using an encoder and decode it to retain original structure using a decoder.


1 Answers

You will probably have to see for yourself which one is better because it depends on the problem you're solving. However, I'm giving you the difference between the two approaches.

Difference <code>return_sequences=True</code> and RepeatVector Essentially, return_sequences=True returns all the outputs the encoder observed in the past, while RepeatVector repeats the very last output of the encoder.

like image 83
thushv89 Avatar answered Oct 05 '22 22:10

thushv89