MultiRNN and static_rnn error: Dimensions must be equal, but are 256 and 129

Question

I want to build an LSTM network with 3 Layers. Here's the code:

num_layers=3
time_steps=10
num_units=128
n_input=1
learning_rate=0.001
n_classes=1
...

x=tf.placeholder("float",[None,time_steps,n_input],name="x")
y=tf.placeholder("float",[None,n_classes],name="y")
input=tf.unstack(x,time_steps,1)

lstm_layer=rnn_cell.BasicLSTMCell(num_units,state_is_tuple=True)
network=rnn_cell.MultiRNNCell([lstm_layer for _ in range(num_layers)],state_is_tuple=True)

outputs,_=rnn.static_rnn(network,inputs=input,dtype="float")

With num_layers=1 it works fine, but with more than one layer I get the error at this line:

outputs,_=rnn.static_rnn(network,inputs=input,dtype="float")

ValueError: Dimensions must be equal, but are 256 and 129 for 'rnn/rnn/multi_rnn_cell/cell_0/cell_0/basic_lstm_cell/MatMul_1' (op: 'MatMul') with input shapes: [?,256], [129,512].

Can anyone explain where the values 129 and 512 are coming from?

Answer 1

You should not reuse the same cell for the first and deeper layers, because their inputs are different, hence kernel matrices are different. Try this:

# Extra function is for readability. No problem to inline it.
def make_cell(lstm_size):
  return tf.nn.rnn_cell.BasicLSTMCell(lstm_size, state_is_tuple=True)

network = rnn_cell.MultiRNNCell([make_cell(num_units) for _ in range(num_layers)], 
                                state_is_tuple=True)