Variational Autoencoder on Timeseries with LSTM in Keras

Question

I am working on a Variational Autoencoder (VAE) to detect anomalies in time series. So far I worked with this tut https://blog.keras.io/building-autoencoders-in-keras.html and this https://wiseodd.github.io/techblog/2016/12/10/variational-autoencoder/.

Still, I have some trouble while implementing the VAE. I have 77093 samples which have 1 dimension. I use timesteps=100 to make predictions. So I reshape my x_train as follows:

x_train.shape = (77093, 100, 1)

The model:

inputs = Input(shape=(timesteps, input_dim))
encoded = LSTM(32)(inputs)

mu = Dense(1, activation='linear')(encoded)
log_sigma = Dense(1, activation='linear')(encoded)

z = Lambda(sample_z)([mu, log_sigma])

decoded = RepeatVector(timesteps)(z)
decoded = LSTM(1, return_sequences=True)(decoded)
decoded = LSTM(1)(decoded)

sequence_autoencoder = Model(inputs, decoded)

I sample from:

def sample_z(args):
  mu, log_sigma = args
  eps = K.random_normal(shape=(50, 1), mean=0., stddev=1.)
 return mu + K.exp(log_sigma / 2) * eps

The model compiles. But I dont know if it is correct.

1.) I dont really understand the RepeatVector Layer and if it is necessary to repeat my sample z. But if I dont use RepeatVector Layer the LSTM-Layer throws an error, because it expects a 3 dim Input.

2.)I am not sore about the dimension reduction in the latent variable. Cause my In_dim=1. What exactly gets reduced?

Thanks in advance.

Answer 1

I have answered your questions below. I would suggest to read a little bit more about LSTMs, e.g. colah's blog post. This will help you understand what it is about, and you will see that your questions are related to the inner workings of an LSTM network.

1) The decoding LSTM network needs something as an input, just as your encoding LSTM used the input data from your dataset. You could either feedback the output of your decoding LSTM, our just repeat the latent state from your encoder (as your code snippet is doing). There are several variations possible, but it seems like most works use the latent vector for initialization of the hidden state in the decoding LSTM, and then feedback the output to the input when rolling out further. (See e.g. Recurrent AE model for multidimensional time series representation and Variational Recurrent Auto-encoders)

2) Your input dimension is 1, but over 100 time steps. Thus your actual input dimension is 100x1. If you choose the dimension of your hidden layer in the LSTM to be 32, than your input effectively gets reduced from 100x1 to 32.

If you still require more information, someone posted a similar question on GitHub.