Keras simple RNN implementation

Question

I found problems when trying to compile a network with one recurrent layer. It seems there is some issue with the dimensionality of the first layer and thus my understanding of how RNN layers work in Keras.

My code sample is:

model.add(Dense(8,
                input_dim = 2,
                activation = "tanh",
                use_bias = False))
model.add(SimpleRNN(2,
                    activation = "tanh",
                    use_bias = False))
model.add(Dense(1,
                activation = "tanh",
                use_bias = False))

The error is

ValueError: Input 0 is incompatible with layer simple_rnn_1: expected ndim=3, found ndim=2

This error is returned regardless of input_dim value. What am I missing ?

Answer 1

That message means: the input going into the rnn has 2 dimensions, but an rnn layer expects 3 dimensions.

For an RNN layer, you need inputs shaped like (BatchSize, TimeSteps, FeaturesPerStep). These are the 3 dimensions expected.

A Dense layer (in keras 2) can work with either 2 or 3 dimensions. We can see that you're working with 2 because you passed an input_dim instead of passing an input_shape=(Steps,Features).

There are many possible ways to solve this, but the most meaningful and logical would be a case where your input data is a sequence with time steps.

Solution 1 - Your training data is a sequence:

If your training data is a sequence, you shape it like (NumberOfSamples, TimeSteps, Features) and pass it to your model. Make sure you use input_shape=(TimeSteps,Features) in the first layer instead of using input_dim.

Solution 2 - You reshape the output of the first dense layer so it has the additional dimension:

model.add(Reshape((TimeSteps,Features)))

Make sure that the product TimeSteps*Features is equal to 8, the output of your first dense layer.