LSTM time sequence generation using PyTorch

Question

For several days now, I am trying to build a simple sine-wave sequence generation using LSTM, without any glimpse of success so far.

I started from the time sequence prediction example

All what I wanted to do differently is:

• Use different optimizers (e.g RMSprob) than LBFGS
• Try different signals (more sine-wave components)

This is the link to my code. "experiment.py" is the main file

What I do is:

• I generate artificial time-series data (sine waves)
• I cut those time-series data into small sequences
• The input to my model is a sequence of time 0...T, and the output is a sequence of time 1...T+1

What happens is:

• The training and the validation losses goes down smoothly
• The test loss is very low
• However, when I try to generate arbitrary-length sequences, starting from a seed (a random sequence from the test data), everything goes wrong. The output always flats out

I simply don't see what the problem is. I am playing with this for a week now, with no progress in sight. I would be very grateful for any help.

Thank you

Answer 1

This is normal behaviour and happens because your network is too confident of the quality of the input and doesn't learn to rely on the past (on it's internal state) enough, relying soley on the input. When you apply the network to its own output in the generation setting, the input to the network is not as reliable as it was in the training or validation case where it got the true input.

I have two possible solutions for you:

• The first is the simplest but less intuitive one: Add a little bit of Gaussian noise to your input. This will force the network to rely more on its hidden state.

• The second, is the most obvious solution: during training, feed it not the true input but its generated output with a certain probability p. Start out training with p=0 and gradually increase it so that it learns to general longer and longer sequences, independently. This is called schedualed sampling, and you can read more about it here: https://arxiv.org/abs/1506.03099 .