In Keras, how to get 3D input and 3D output for LSTM layers

Question

In my original setting, I got

X1 = (1200,40,1)
y1 = (1200,10)

Then, I work perfectly with my codes:

model = Sequential()
model.add(LSTM(12, input_shape=(40, 1), return_sequences=True))
model.add(LSTM(12, return_sequences=True))
model.add(LSTM(6, return_sequences=False))
model.add((Dense(10)))

Now, I further got another time series data same sizes as X1 and y1. i.e.,

X2 = (1200,40,1)
y2 = (1200,10)

Now, I stack X1, X2 and y1, y2 as 3D arrays:

X_stack = (1200,40,2)
y_stack = (1200,10,2)

Then, I try to modify my keras code like:

model = Sequential()
model.add(LSTM(12, input_shape=(40, 2), return_sequences=True))
model.add(LSTM(12, return_sequences=True))
model.add(LSTM(6, return_sequences=False))
model.add((Dense((10,2))))

I want my code work directly with the 3D arrays X_stack and y_stack without reshaping them as 2D arrays. Would you give me a hand on how to modify the settings? Thank you.

Answer 1

i can not give a short answer to this question however i think there is clarification needed about some basic concepts of LSTM (one-to-one, one-to-many,...)

As a superstructure RNNs (including LSTMs) are sequential, they are constructed to find time-like correlations, while CNNs are spatial they are build to find space-like correlations

Then there is a further differentiation of LSTM in one-to-one, one-to-many, many-to-one and many-to-many like shown in Many to one and many to many LSTM examples in Keras

The network type that is wanted here is point 5 in Many to one and many to many LSTM examples in Keras and it says :

Many-to-many when number of steps differ from input/output length: this is freaky hard in Keras. There are no easy code snippets to code that.

It is type 5 because input shape is X_stack = (1200,40,2) and output shape is y_stack = (1200,10,2) so the number of timesteps differ (40 input and 10 output)

If you could manage to have an equal number of input and output timesteps you can reshape input and output data (numpy.reshape) like in keras LSTM feeding input with the right shape ( note the arrangement of the [ and ] in the arrays). This does not mean reshaping to 2D ( i.e. flattening ). In https://machinelearningmastery.com/timedistributed-layer-for-long-short-term-memory-networks-in-python/ is a complete example for building a many-to-many LSTM with equal input and output timesteps using TimeDistributed layer

Only for completeness, for spatio-temporal data there are also CNN-LSTMs however this does not apply here because two stacked timeseries have no explicit spatial correlations :

If you have a 3D quantity, i.e. a distribution in a volume that changes over time and want to learn this then you have to use a CNN-LSTM network. In this approach both the 3D information and the temporal information is preserved. With 3D information is preserved is meant that the spatial information is not discarded. Normally in time-like learners like LSTM this spatial information is often discarded i.e. by flattening an image before processing it in an LSTM. A complete tutorial how a (spatio-temporal) CNN-LSTM can be built in keras is in https://machinelearningmastery.com/cnn-long-short-term-memory-networks/

Answer 2

I am assuming that there is an error somewhere in the shapes that you reported for your arrays. I'm guessing y_stack.shape == (1200, 10, 2), is that correct?

However, here is one possibility to do what you describe:

model = Sequential()
model.add(LSTM(12, input_shape=(40, 2), return_sequences=True))
model.add(LSTM(12, return_sequences=True))
model.add(LSTM(6, return_sequences=False))
model.add(Dense(10 * 2))
model.add(Reshape((10, 2)))

The output of the network is created as a 2D tensor by the Dense layer, and then reshaped to a 3D tensor by the Reshape. From an input-output perspective, this should behave like you specified.