Menu
After you pass a video frame through a convnet and get an output feature map, how do you pass that data into an LSTM? Also, how do you pass multiple frames to the LSTM thru the CNN?
In other works I want to process video frames with an CNN to get the spatial features. Then I want pass these features to an LSTM to do temporal processing on the spatial features. How do I connect the LSTM to the video features? For example if the input video is 56x56 and then when passed through all of the CNN layers, say it comes out as 20: 5x5's. How are these connected to the LSTM on a frame by frame basis? ANd shoudl they go through a fully connected layer first?
Thanks, Jon
971
A CNN LSTM can be defined by adding CNN layers on the front end followed by LSTM layers with a Dense layer on the output. It is helpful to think of this architecture as defining two sub-models: the CNN Model for feature extraction and the LSTM Model for interpreting the features across time steps.
The ConvLSTM differs from simple CNN + LSTM in that, for CNN + LSTM, the convolution structure (CNN) is applied as the first layer and sequentially LSTM layer is applied in the second layer.
An LSTM is designed to work differently than a CNN because an LSTM is usually used to process and make predictions given sequences of data (in contrast, a CNN is designed to exploit “spatial correlation” in data and works well on images and speech).
This is called the CNN LSTM model, specifically designed for sequence prediction problems with spatial inputs, like images or videos. This architecture involves using Convolutional Neural Network (CNN) layers for feature extraction on input data combined with LSTMs to perform sequence prediction on the feature vectors.
Basically, you can flatten each frame features and feed them into one LSTM cell. With CNN, it's the same. You can feed each output of CNN into one LSTM cell.
For FC, it's up to you.
See a network structure from http://www.eecs.berkeley.edu/Pubs/TechRpts/2014/EECS-2014-180.pdf.
195
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
