How to average summaries over multiple batches?

Question

Assuming I have a bunch of summaries defined like:

loss = ...
tf.scalar_summary("loss", loss)
# ...
summaries = tf.merge_all_summaries()

I can evaluate the summaries tensor every few steps on the training data and pass the result to a SummaryWriter. The result will be noisy summaries, because they're only computed on one batch.

However, I would like to compute the summaries on the entire validation dataset. Of course, I can't pass the validation dataset as a single batch, because it would be too big. So, I'll get summary outputs for each validation batch.

Is there a way to average those summaries so that it appears as if the summaries have been computed on the entire validation set?

Answer 1

Do the averaging of your measure in Python and create a new Summary object for each mean. Here is what I do:

accuracies = []

# Calculate your measure over as many batches as you need
for batch in validation_set:
  accuracies.append(sess.run([training_op]))

# Take the mean of you measure
accuracy = np.mean(accuracies)

# Create a new Summary object with your measure
summary = tf.Summary()
summary.value.add(tag="%sAccuracy" % prefix, simple_value=accuracy)

# Add it to the Tensorboard summary writer
# Make sure to specify a step parameter to get nice graphs over time
summary_writer.add_summary(summary, global_step)

Answer 2

I would avoid calculating the average outside the graph.

You can use tf.train.ExponentialMovingAverage:

ema = tf.train.ExponentialMovingAverage(decay=my_decay_value, zero_debias=True)
maintain_ema_op = ema.apply(your_losses_list)

# Create an op that will update the moving averages after each training step.
with tf.control_dependencies([your_original_train_op]):
    train_op = tf.group(maintain_ema_op)

Then, use:

sess.run(train_op)

That will call maintain_ema_op because it is defined as a control dependency.

In order to get your exponential moving averages, use:

moving_average = ema.average(an_item_from_your_losses_list_above)

And retrieve its value using:

value = sess.run(moving_average)

This calculates the moving average within your calculation graph.

Answer 3

I think it's always better to let tensorflow do the calculations.

Have a look at the streaming metrics. They have an update function to feed the information of your current batch and a function to get the averaged summary. It's going to look somewhat like this:

accuracy = ... 
streaming_accuracy, streaming_accuracy_update = tf.contrib.metrics.streaming_mean(accuracy)
streaming_accuracy_scalar = tf.summary.scalar('streaming_accuracy', streaming_accuracy)

# set up your session etc. 

for i in iterations:
      for b in batches:
               sess.run([streaming_accuracy_update], feed_dict={...})

     streaming_summ = sess.run(streaming_accuracy_scalar)
     writer.add_summary(streaming_summary, i)

Also see the tensorflow documentation: https://www.tensorflow.org/versions/master/api_guides/python/contrib.metrics

and this question: How to accumulate summary statistics in tensorflow

Answer 4

You can average store the current sum and recalculate the average after each batch, like:

loss_sum = tf.Variable(0.)
inc_op = tf.assign_add(loss_sum, loss)
clear_op = tf.assign(loss_sum, 0.)
average = loss_sum / batches
tf.scalar_summary("average_loss", average)

sess.run(clear_op)
for i in range(batches):
    sess.run([loss, inc_op])

sess.run(average)