How can I access the weights of a recurrent cell in Tensorflow?

Question

One way to improve stability in deep Q-learning tasks is to maintain a set of target weights for the network that update slowly and are used for calculating Q-value targets. As a result at different times in the learning procedure, two different sets of weights are used in the forward pass. For normal DQN this is not difficult to implement, as the weights are tensorflow variables that can be set in a feed_dict ie:

sess = tf.Session()
input = tf.placeholder(tf.float32, shape=[None, 5])
weights = tf.Variable(tf.random_normal(shape=[5,4], stddev=0.1)
bias = tf.Variable(tf.constant(0.1, shape=[4])
output = tf.matmul(input, weights) + bias
target = tf.placeholder(tf.float32, [None, 4])
loss = ...

...

#Here we explicitly set weights to be the slowly updated target weights
sess.run(output, feed_dict={input: states, weights: target_weights, bias: target_bias})

# Targets for the learning procedure are computed using this output.

....

#Now we run the learning procedure, using the most up to date weights,
#as well as the previously computed targets
sess.run(loss, feed_dict={input: states, target: targets})

I'd like to use this target network technique in a recurrent version of DQN, but I don't know how to access and set the weights used inside a recurrent cell. Specifically I'm using a tf.nn.rnn_cell.BasicLSTMCell, but I'd like to know how to do this for any type of recurrent cell.

Answer 1

The BasicLSTMCell does not expose its variables as part of its public API. I recommend that you either look up what names these variables have in your graph and feed those names (those names are unlikely to change since they are in the checkpoints and changing these names would break checkpoint compatibility).

Alternatively, you can make a copy of BasicLSTMCell which does expose the variables. This is the cleanest approach, I think.