Get the loss that a given optimizer is minimizing in Tensorflow

Question

I am working in a unit test system for my tensorflow workspace and I would like to know if there is any method or attribute, given a graph with an optimizer operation (after calling .minimize()), to obtain the final loss tensor that it is optimizing and the variables that it controls.

For example if I call train_op = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy) I would like to retrieve cross_entropy only having access to train_op.

I have access to the train_op object, I only want to know to which loss it is referenced and which variables controls.

Answer 1

Quite trivially:

def build_graph():
  cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(...)
  train_op = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
  return cross_entropy, train_op    # both tensorflow OPs

cross_entropy, train_op = build_graph()

with tf.Session() as sess:
  sess.run(tf.global_variables_initializer())

  # Evaluate and return cross_entropy, and execute the train_op (update the weights)
  result = sess.run([cross_entropy, train_op], feed_dict={...})

  print(result[0])   # The value of the cross entropy loss function

There are many excellent tutorials here: https://github.com/aymericdamien/TensorFlow-Examples

You'll find them doing exactly this in full working models.

If you don't have access to the tensor, then you can look it up in the graph by name as such:

tf.get_default_graph().get_tensor_by_name("example:0")

See this question: Tensorflow: How to get a tensor by name?

Note that if you didn't name your tensors well, this is going to be a royal pain in the rear, so, it's one of many good reasons to name your tensors well. The default name of a tensor will use some reference to the operation, colon, an index number, such as "add:2" for the 3rd add operation.

You can get a list of all tensors in a graph with:

[n.name for n in tf.get_default_graph().as_graph_def().node]

That code is copied from from this question: In Tensorflow, get the names of all the Tensors in a graph

---

Responding to this follow up question in comments:

I would like to know which one is the train_op optimizing without having to name them with a specific name. So given a train_op object, is there any way of retrieving the tensor (or the name of the tensor) which represents the last value that train_op is minimizing? I need it because I am automatizing a set of unit tests so that if I plug a tensorflow model to my system it automatically finds, given the optimizers, the tensors that represent the loss (that way I can automatically perform gradient checks).

I have coded a gradient descent optimizer as part of my research. Here are a few ideas you might consider:

1) Here's a link to the optimizer I followed when I did the same: https://github.com/openai/iaf/blob/master/tf_utils/adamax.py That's implementing AdaMax in python. You'll be interested in _apply_dense() which takes a gradient and its variable and performs the update. It's called for each trainable variable. Note that most optimizers in tensorflow are coded in C, not using the python interface. So I'm not sure if this will help or not, but understanding the process better can't be a bad thing.

2) You can get the gradient of any variable with respect to any other variable. So you could grab the collection of trainable variables with tf.trainable_variables() and then call tf.gradients to get the gradients of the trainable variables with respect to the loss function. You would need the loss function for this rather than the train OP though. I expect you can find the loss automatically from the optimizer.

If you're just trying to find the loss function from the train OP you might find what you need by following the graph dependencies as is described in this question: How can I list all Tensorflow variables a node depends on?

Here's a way I've used before to get a list of each variable and its inputs and outputs. I suspect you could figure out how to traverse this datastructure to find what you need.

tf.get_default_graph().as_graph_def()

Out[6]: 
node {
  name: "x"
  op: "Const"
  attr {
    key: "dtype"
    value {
      type: DT_FLOAT
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_FLOAT
        tensor_shape {
        }
        float_val: 42.0
      }
    }
  }
}