I have a TensorFlow model, and one part of this model evaluates the accuracy. The accuracy
is just another node in the tensorflow graph, that takes in logits
and labels
.
When I want to plot the training accuracy, this is simple: I have something like:
tf.scalar_summary("Training Accuracy", accuracy)
tf.scalar_summary("SomethingElse", foo)
summary_op = tf.merge_all_summaries()
writer = tf.train.SummaryWriter('/me/mydir/', graph=sess.graph)
Then, during my training loop, I have something like:
for n in xrange(1000):
...
summary, ..., ... = sess.run([summary_op, ..., ...], feed_dict)
writer.add_summary(summary, n)
...
Also inside that for loop, every say, 100 iterations, I want to evaluate the validation accuracy. I have a separate feed_dict for this, and I am able to evaluate the validation accuracy very nicely in python.
However, here is my problem: I want to make another summary for the validation accuracy, by using the accuracy
node. I am not clear on how to do this though. Since I have the accuracy
node it makes sense that I should be able to re-use it, but I am unsure how to do this exactly, such that I can also get the validation accuracy written out as a separate scalar_summary...
How might this be possible?
Training accuracy means that identical images are used both for training and testing, while test accuracy represents that the trained model identifies independent images that were not used in training.
You can do this by setting the validation_split argument on the fit() function to a percentage of the size of your training dataset. For example, a reasonable value might be 0.2 or 0.33 for 20% or 33% of your training data held back for validation.
Especially if the dataset split is not random (in case where temporal or spatial patterns exist) the validation set may be fundamentally different, i.e less noise or less variance, from the train and thus easier to to predict leading to higher accuracy on the validation set than on training.
You can reuse the the accuracy node but you need to use two different SummaryWriters, one for the training runs and one for the test data. Also you have to assign the scalar summary for accuracy to a variable.
accuracy_summary = tf.scalar_summary("Training Accuracy", accuracy)
tf.scalar_summary("SomethingElse", foo)
summary_op = tf.merge_all_summaries()
summaries_dir = '/me/mydir/'
train_writer = tf.train.SummaryWriter(summaries_dir + '/train', sess.graph)
test_writer = tf.train.SummaryWriter(summaries_dir + '/test')
Then in your training loop you have the normal training and record your summaries with the train_writer. In addition you run the graph on the test set each 100th iteration and record only the accuracy summary with the test_writer.
# Record train set summaries, and train
summary, _ = sess.run([summary_op, train_step], feed_dict=...)
train_writer.add_summary(summary, n)
if n % 100 == 0: # Record summaries and test-set accuracy
summary, acc = sess.run([accuracy_summary, accuracy], feed_dict=...)
test_writer.add_summary(summary, n)
print('Accuracy at step %s: %s' % (n, acc))
You can then point TensorBoard to the parent directory (summaries_dir) and it will load both data sets.
This can be also found in the TensorFlow HowTo's https://www.tensorflow.org/versions/r0.11/how_tos/summaries_and_tensorboard/index.html
To run the same operation but get summaries with different feed_dict data, simply attach two summary ops to that op. Say you want to run accuracy op on both validation and test data and want to get summaries for both:
validation_acc_summary = tf.summary.scalar('validation_accuracy', accuracy) # intended to run on validation set
test_acc_summary = tf.summary.scalar('test_accuracy', accuracy) # intended to run on test set
with tf.Session() as sess:
# do your thing
# ...
# accuracy op just needs labels y_ and input x to compute logits
validation_summary_str = sess.run(validation_acc_summary, feed_dict=feed_dict={x: mnist.validation.images,y_: mnist.validation.labels})
test_summary_str = sess.run(test_acc_summary, feed_dict={x: mnist.test.images,y_: mnist.test.labels})
# assuming you have a tf.summary.FileWriter setup
file_writer.add_summary(validation_summary_str)
file_writer.add_summary(test_summary_str)
Also remember you can always pull raw (scalar) data out of the protobuff summary_str like this and do your own logging.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With