Keras LSTM Time Series

Question

I have a problem and at this point I'm completely lost as to how to solve it. I'm using Keras with an LSTM layer to project a time series. I'm trying to use the previous 10 data points to predict the 11th.

Here's the code:

from keras.models import Sequential
from keras.layers.core import Dense, Activation, Dropout
from keras.layers.recurrent import LSTM

def _load_data(data):
"""
data should be pd.DataFrame()
"""
n_prev = 10
docX, docY = [], []
for i in range(len(data)-n_prev):
    docX.append(data.iloc[i:i+n_prev].as_matrix())
    docY.append(data.iloc[i+n_prev].as_matrix())
if not docX:
    pass
else:
    alsX = np.array(docX)
    alsY = np.array(docY)
    return alsX, alsY

X, y = _load_data(df_test)

X_train = X[:25]
X_test = X[25:]

y_train = y[:25]
y_test = y[25:]

in_out_neurons = 2
hidden_neurons = 300
model = Sequential()
model.add(LSTM(in_out_neurons, hidden_neurons, return_sequences=False))
model.add(Dense(hidden_neurons, in_out_neurons))
model.add(Activation("linear"))
model.compile(loss="mean_squared_error", optimizer="rmsprop")
model.fit(X_train, y_train, nb_epoch=10, validation_split=0.05)

predicted = model.predict(X_test)

So I'm taking the input data (a two column dataframe), creating X which is an n by 10 by 2 array, and y which is an n by 2 array which is one step ahead of the last row in each array of X (labeling the data with the point directly ahead of it.

predicted is returning

[[ 7.56940445,  5.61719704],
[ 7.57328357,  5.62709032],
[ 7.56728049,  5.61216415],
[ 7.55060187,  5.60573629],
[ 7.56717342,  5.61548522],
[ 7.55866942,  5.59696181],
[ 7.57325984,  5.63150951]]

but I should be getting

[[ 73,  48],
[ 74,  42],
[ 91,  51],
[102,  64],
[109,  63],
[ 93,  65],
[ 92,  58]]

The original data set only has 42 rows, so I'm wondering if there just isn't enough there to work with? Or am I missing a key step in the modeling process maybe? I've seen some examples using Embedding layers etc, is that something I should be looking at?

Thanks in advance for any help!

Answer 1

Hey Ryan!

I know it's late but I just came across your question hope it's not too late or that you still find some knowledge here.

First of all, Stackoverflow may not be the best place for this kind of question. First reason to that is you have a conceptual question that is not this site's purpose. Moreover your code runs so it's not even a matter of general programming. Have a look at stats.

Second from what I see there is no conceptual error. You're using everything necessary that is:

• lstm with propper dimensions
• return_sequences=false just before your Dense layer
• linear activation for your output
• mse cost/loss/objective function

Third I however find it extremely unlikely that your network learns anything with so few pieces of data. You have to understand that you have less data than parameters here! For the great majority of supervised learning algorithm, the first thing you need is not a good model, it's good data. You can not learn from so few examples, especially not with a complex model such as LSTM networks.

Fourth It seems like your target data is made of relatively high values. First step of pre-processing here could be to standardize the data : center it around zero - that is translate your data by its mean - and rescale by ists standard deviation. This really helps learning!

Fifth In general here are a few things you should look into to improve learning and reduce overfitting :

• Dropout
• Batch Normalization
• Other optimizer (such as Adam)
• Gradient clipping
• Random hyper parameter search
• (This is not exhaustive, if you're reading this and think something should be added, comment it so it's useful for future readers!)

Last but NOT least I suggest you look at this tutorial on Github, especially the recurrent tutorial for time series with keras.

PS: Daniel Hnyk updated his post ;)