LSTM with Condition

Question

I'm studying LSTM with CNN in tensorflow. I want to put some scalar label into LSTM network as a condition. Does anybody know which LSTM is what I meant? If available, please let me know the usage of that

Thank you.

Answer 1

This thread might interest you: Adding Features To Time Series Model LSTM.

You have basically 3 possible ways:

Let's take an example with weather data from two different cities: Paris and San Francisco. You want to predict the next temperature based on historical data. But at the same time, you expect the weather to change based on the city. You can either:

• Combine the auxiliary features with the time series data, at the beginning or at the end (ugly!).
• Concatenate the auxiliary features with the output of the RNN layer. It's some kind of post-RNN adjustment since the RNN layer won't see this auxiliary info.
• Or just initialize the RNN states with a learned representation of the condition (e.g. Paris or San Francisco).

I wrote a library to condition on auxiliary inputs. It abstracts all the complexity and has been designed to be as user-friendly as possible:

https://github.com/philipperemy/cond_rnn/

The implementation is in tensorflow (>=1.13.1) and Keras.

Hope it helps!

Answer 2

Heres an example of applying CNN and LSTM over the output probabilities of a sequence, like you asked:

def build_model(inputs):

    BATCH_SIZE = 4
    NUM_CLASSES = 2
    NUM_UNITS = 128
    H = 224
    W = 224
    C = 3
    TIME_STEPS = 4
    # inputs is assumed to be of shape (BATCH_SIZE, TIME_STEPS, H, W, C)
    # reshape your input such that you can apply the CNN for all images
    input_cnn_reshaped = tf.reshape(inputs, (-1, H, W, C))

    # define CNN, for instance vgg 16
    cnn_logits_output, _ = vgg_16(input_cnn_reshaped, num_classes=NUM_CLASSES)
    cnn_probabilities_output = tf.nn.softmax(cnn_logits_output)

    # reshape back to time series convention
    cnn_probabilities_output = tf.reshape(cnn_probabilities_output, (BATCH_SIZE, TIME_STEPS, NUM_CLASSES))

    # perform LSTM over the probabilities per image
    cell = tf.contrib.rnn.LSTMCell(NUM_UNITS)
    _, state = tf.nn.dynamic_rnn(cell, cnn_probabilities_output)

    # employ FC layer over the last state
    logits = tf.layers.dense(state, NUM_UNITS)

    # logits is of shape (BATCH_SIZE, NUM_CLASSES)
    return logits

By the way, a better approach would be to employ the LSTM over the last hidden layer, i.e to use the CNN as feature extractor and make the prediction over sequences of features.