Is it possible to get the objective function value during each training step?

Question

In the usual TensorFlow training loop, such as

train_op = tf.train.AdamOptimizer().minimize(cross_entropy)

with tf.Session() as sess:
    for i in range(num_steps):
        # ...
        train_op.run(feed_dict = feed_dict)

train_op.run returns None.

However, sometimes it's useful to collect intermediate results, such as the value of the objective or the accuracy.

Adding extra sess.run calls would require doing the forward propagation again, increasing the run time:

train_op = tf.train.AdamOptimizer().minimize(cross_entropy)

with tf.Session() as sess:
    for i in range(num_steps):
        # ...
        o, a = sess.run([objective, accuracy], feed_dict = feed_dict)
        train_op.run(feed_dict = feed_dict)

Is it possible to do this in TensorFlow in one go?

---

Edit:

People suggested

sess.run([objective, accuracy, train_op], feed_dict = feed_dict)

but the results depend on the execution order of the list elements:

[objective, accuracy, train_op]

which appears to be undefined -- you get different results depending on whether CUDA is being used.

Answer 1

Simply add you train_op to the list of nodes to be evaluated.

o, a, _ = sess.run([objective, accuracy, train_op], feed_dict = feed_dict)

Regarding the training step and its order in the evaluation, I made the following small experiment:

import tensorflow as tf
x = tf.Variable(0, dtype=tf.float32)
loss = tf.nn.l2_loss(x-1)
train_opt = tf.train.GradientDescentOptimizer(1)
train_op = train_opt.minimize(loss)
init_op = tf.global_variables_initializer()

sess = tf.Session()
sess.run(init_op)
x_val, _, loss_val = sess.run([x, train_op, loss])
# returns x_val = 1.0, loss_val = 0.5

The situation is more confused than I initially thought. What seems to be a given is that the order of execution of the fetches does not depend of their respective position in the list: x_val and loss_val will be the same independently of their position in the list.

However, as @MaxB noticed, their order of execution is not guaranteed. When running the above code on GPU, x_val is set to 0.0, the initial value. However, when running on CPU, x_val is 1.0, that is, the value after the update from train_op.

This configuration-dependant behavior could be limited to variables updated by training operations, as the experiment above suggests, but their is no guarantee coming from tf's documentation.

Answer 2

You can provide as many ops as you want in sess.run. In your case you use objective and accuracy. Add your train_op there. Results of it is not needed so you can use _. Basically:

o, a, _ = sess.run([objective, accuracy, train_op], feed_dict = feed_dict)

P.S. regarding your comment, sess.run will not run the graph 3 times. ALso it will not necessarily even will run the graph once. It will figure out all ops that should be evaluated to evaluate 3 things you provided and will run all these ops (thus running a subgraph once)