Convolution2D + LSTM versus ConvLSTM2D

Question

Are 1 and 2 the same?

1. Use Convolution2D layers and LSTM layers
2. Use ConvLSTM2D

If there is any difference, could you explain it for me?

Answer 1

They are not exactly the same, here is why:

1. Use Convolution2D layers and LSTM layers

As it is known, Convolution2D serves well for capturing image or spatial features, whilst LSTM are used to detect correlations over time. However, by stacking these kind of layers, the correlation between space and time features may not be captured properly.

2. Use ConvLSTM2D

To solve this, Xingjian Shi et al. proposed a network structure able to capture spatiotemporal correlations, namely ConvLSTM. In Keras, this is reflected in the ConvLSTM2D class, which computes convolutional operations in both the input and the recurrent transformations.

Keras code

Too illustrate this, you can see here the LSTM code, if you go to the call method from LSTMCell, you'd only see:

    x_i = K.dot(inputs_i, self.kernel_i)     x_f = K.dot(inputs_f, self.kernel_f)     x_c = K.dot(inputs_c, self.kernel_c)     x_o = K.dot(inputs_o, self.kernel_o) 

Instead, the ConvLSTM2DCell class calls:

    x_i = self.input_conv(inputs_i, self.kernel_i, self.bias_i, padding=self.padding)     x_f = self.input_conv(inputs_f, self.kernel_f, self.bias_f, padding=self.padding)     x_c = self.input_conv(inputs_c, self.kernel_c, self.bias_c, padding=self.padding)     x_o = self.input_conv(inputs_o, self.kernel_o, self.bias_o, padding=self.padding)     h_i = self.recurrent_conv(h_tm1_i, self.recurrent_kernel_i)     h_f = self.recurrent_conv(h_tm1_f, self.recurrent_kernel_f)     h_c = self.recurrent_conv(h_tm1_c, self.recurrent_kernel_c)     h_o = self.recurrent_conv(h_tm1_o, self.recurrent_kernel_o) 

Where:

def input_conv(self, x, w, b=None, padding='valid'):     conv_out = K.conv2d(x, w, strides=self.strides,                         padding=padding,                         data_format=self.data_format,                         dilation_rate=self.dilation_rate)     if b is not None:         conv_out = K.bias_add(conv_out, b,                               data_format=self.data_format)     return conv_out  def recurrent_conv(self, x, w):     conv_out = K.conv2d(x, w, strides=(1, 1),                         padding='same',                         data_format=self.data_format)     return conv_out 

In LSTM, the equivalent for h_x (recurrent transformations) would be:

K.dot(h_tm1_x, self.recurrent_kernel_x) 

Instead of ConvLSTM2D's:

self.recurrent_conv(h_tm1_x, self.recurrent_kernel_x) 

These kind of transformations could not be computed with stacked Conv2D and LSTM layers.

Answer 2

1. Use Convolution2D layers and LSTM layer

In this technique, you stack convolution and LSTM layers. The convolutional layers help you to learn the spatial features and the LSTM helps you learn the correlation in time.

2.Use ConvLSTM2D

ConvLSTM is a LSTM in which the gates (input to state and state to state transitions) are convolution operations.
Research paper- Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting

More about ConvLSTM in this SO answer