Structure of initial state for stacked LSTM

Question

What is the required structure for an initial state on a multilayer/stacked RNN in TensorFlow (1.13.1) using the tf.keras.layers.RNN API?

I tried the following:

lstm_cell_sizes = [256, 256, 256]
lstm_cells = [tf.keras.layers.LSTMCell(size) for size in lstm_cell_sizes]

state_init = [tf.placeholder(tf.float32, shape=[None] + cell.state_size) for cell in lstm_cells]

tf.keras.layers.RNN(lstm_cells, ...)(inputs, initial_state=state_init)

This results in:

ValueError: Could not pack sequence. Structure had 6 elements, but flat_sequence had 3 elements.  Structure: ([256, 256], [256, 256], [256, 256]), flat_sequence: [<tf.Tensor 'player/Placeholder:0' shape=(?, 256, 256) dtype=float32>, <tf.Tensor 'player/Placeholder_1:0' shape=(?, 256, 256) dtype=float32>, <tf.Tensor 'player/Placeholder_2:0' shape=(?, 256, 256) dtype=float32>].

If I change state_init to be a flattened list of tensors with shape [None, 256] instead, I am getting:

ValueError: An `initial_state` was passed that is not compatible with `cell.state_size`. Received `state_spec`=[InputSpec(shape=(None, 256), ndim=2), InputSpec(shape=(None, 256), ndim=2), InputSpec(shape=(None, 256), ndim=2)]; however `cell.state_size` is [[256, 256], [256, 256], [256, 256]]

The Tensorflow RNN docs are fairly vague on this:

"You can specify the initial state of RNN layers symbolically by calling them with the keyword argument initial_state. The value of initial_state should be a tensor or list of tensors representing the initial state of the RNN layer."

Answer 1

I believe this how you do it in TF2:

import tensorflow.compat.v2 as tf #If you have a newer version of TF1
#import tensorflow as tf          #If you have TF2

sentence_max_length = 5
batch_size = 3
n_hidden = 2
x = tf.constant(np.reshape(np.arange(30),(batch_size,sentence_max_length, n_hidden)), dtype = tf.float32)

stacked_lstm = tf.keras.layers.StackedRNNCells([tf.keras.layers.LSTMCell(128) for _ in range(2)])

lstm_layer = tf.keras.layers.RNN(stacked_lstm,return_state=False,return_sequences=False)

result = lstm_layer(x)
print(result)