Formatting inputs for LSTM layer with variable timestep using Tensorflow

Question

According to documentation, the LSTM layer should handle inputs with (None, CONST, CONST) shape. For variable timestep, it should be able to handle inputs with (None, None, CONST) shape.

Let say my data is the following :

X = [
    [
        [1, 2, 3],
        [4, 5, 6]
    ],
    [
        [7, 8, 9]
    ]
]
Y = [0, 1]

And my model :

model = tf.keras.models.Sequential([
    tf.keras.layers.LSTM(32, activation='tanh',input_shape=(None, 3)),
    tf.keras.layers.Dense(1, activation='sigmoid')
])
model.compile(loss='categorical_crossentropy', optimizer='adam')
model.fit(X, Y)

My question is: how should I format these inputs to make this code work ?

I cannot use pandas dataframes here, as I was used to. If I run the code above, I get this error :

Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 1 array(s), but instead got the following list of 2 arrays:

And if I change the last line with :

model.fit(np.array(X), np.array(Y))

The error is now :

Error when checking input: expected lstm_8_input to have 3 dimensions, but got array with shape (2, 1)

Answer 1

You are close but in Keras / Tensorflow you need to pad your sequences and then use a Masking to let the LSTM skip those padded ones. Why? Because your entries in your tensor need to have the same shape (batch_size, max_length, features). So if you have variable length, the sequence gets padded.

You can use keras.preprocessing.sequence.pad_sequences to pad your sequences to obtain something like:

X = [
    [
        [1, 2, 3],
        [4, 5, 6]
    ],
    [
        [7, 8, 9],
        [0, 0, 0],
    ]
]
X.shape == (2, 2, 3)
Y = [0, 1]
Y.shape == (2, 1)

And then use the masking layer:

model = tf.keras.models.Sequential([
    tf.keras.layers.Masking(), # this tells LSTM to skip certain timesteps
    tf.keras.layers.LSTM(32, activation='tanh',input_shape=(None, 3)),
    tf.keras.layers.Dense(1, activation='sigmoid')
])
model.compile(loss='binary_crossentropy', optimizer='adam')
model.fit(X, Y)

You also want binary_crossentropy since you have a binary classification problem with sigmoid output.