Predictions using a Keras Recurrent Neural Network - accuracy is always 1.0

Question

TLDR: How do I use a Keras RNN to predict the next value in a sequence?

---

I have a list of sequential values. I want to feed them into a RNN to predict the next value in the sequence.

[ 0.43589744  0.44230769  0.49358974 ...,  0.71153846  0.70833333 0.69230769]

I'm using Keras to do this and can get a network with a decreasing loss but the accuracy is consistently 1.0. This is wrong. y_tests != model.predict(x_tests).

Epoch 0
1517/1517 [==============================] - 0s - loss: 0.0726 - acc: 1.0000 - val_loss: 0.0636 - val_acc: 1.0000
Epoch 1
1517/1517 [==============================] - 0s - loss: 0.0720 - acc: 1.0000 - val_loss: 0.0629 - val_acc: 1.0000
...

Here's my network.

model = Sequential()
model.add(SimpleRNN(1, 100))
model.add(Dense(100, 1, activation = "sigmoid"))
model.compile(loss="mean_squared_error", optimizer = "sgd")

I have tried a SimpleRNN, GRU and LSTM but have had no luck. Here is how the data is formatted.

# Current value
y_train = [[ 0.60576923] [ 0.64102564] [ 0.66025641] ..., [ 0.71153846] [ 0.70833333] [ 0.69230769]]

# Previous 10 values
x_train_10 = [
    [[ 0.65064103] [ 0.66346154] [ 0.66346154] ..., [ 0.72115385] [ 0.72435897] [ 0.71153846]] ...,
    [[ 0.66346154] [ 0.66346154] [ 0.67628205] ..., [ 0.72435897] [ 0.71153846] [ 0.70833333]]
]

# Previous value
x_train_1 = [[ 0.58333333] [ 0.60576923] [ 0.64102564] ...,  [ 0.72435897] [ 0.71153846] [ 0.70833333]]

# So here are the shapes...
y_train.shape    = (1895, 1)
x_train_10.shape = (1895, 10, 1)
x_train_1.shape  = (1895, 1)

Each element in x_train_10 is a list of the previous 10 values. I formatted it like this to follow Keras's documentation that recurrent layers take inputs of shape (nb_samples, timesteps, input_dim).

I have also tried using an Embedding layer with no luck. (This may be the wrong way to use it - I've only seen it used in classification not prediction).

model = Sequential()
model.add(Embedding(1, 30))
model.add(LSTM(30, 100))
...

pad_sequences also did not work.

x_train_1 = sequence.pad_sequences(x_train_1, maxlen = None, dtype = "float32")

---

I want to get the RNN working with this simple data/architecture so I can use it for more complex problems later.

Thanks :)

Answer 1

I posted a similar question on the Keras Github page and got a good answer.

---

lukedeo said that acc: 1.0000 means that both the true output and the predicted output are greater than 0.5 or vice versa. Instead, I should look at loss, or mse, to determine the accuracy of the model. This is because my network is a regression not a classifier/clusterer.

Root mean squared error is a good measure of accuracy. accuracy_percent = 1 - np.sqrt(mse)

---

fchollet (the Keras creator) elaborated by saying that "accuracy is not relevant at all for a regression problem."

When doing a classification problem, accuracy can be made relevant by setting class_mode to 'categorical' or 'binary' in model.comple(...) depending on the target (network output).