Access internal forget gate value in lstm node

Question

I have currently created an LSTM network using Keras and have to get the internal forget gate values for each node of the network. I have to get the forget gate/values everytime the predict() function is called. is it possible to do so?

If not then is there any other library which gives access to these internal gate values in a convenient manner at every step?

Looking forward to some help on this at the earliest. Thank you.

Answer 1

How to get internal weights and parameters?

If you are using Keras with Tensorflow backend (which is the recommended backend), you can access to the internal layer values (like weights & biases) using tf.global_variables(). For convenience, create a dictionary which map the layer names to the corresponding tf.Variable:

variable_dict = dict([])
for variable in tf.global_variable():
    variable_dict[variable.name] = variable

After this simple mapping, you can refer to a variable directly by its name. The LSTM layer of Keras (let's say, the model.summary() shows that the LSTM layer is named as 'lstm_1') has the following internal variables: 'lstm_1/kernel:0', 'lstm_1/recurrent_kernel:0', and 'lstm_1/bias:0'. To get their values, you will need to have some understanding of Session and Tensor objects of Tensorflow.

Now is the tricky part: how does Keras store the variables in these tensors? The documentation does not specify any information about it, so unfortunately you will have to refer to the source code. Look at the LSTMCell class, especially the build (describes which variables are defined) and call (describes how these variables are used in calculation) methods. Seems like the internal parameters are stored in the variables, named as *_i, *_f, *_c, *_o (e.g. self.kernel_o).

---

How to get in-between values during calculation?

Unfortunately, you will have to directly modify the calculation graph. It is easier than it sounds: just edit the Tensorflow's code and store these variables somewhere in a named tensor in the global scope.

---

If not then is there any other library which gives access to these internal gate values in a convenient manner at every step?

As you see, Keras with a Tensorflow backend is very convenient. I don't think that you can get any more convenient than that.