How to display per-node memory in TensorBoard?

Question

In another question, someone showed a screenshot of his TensorBoard, displaying memory usage per node:

I never see those in my experiments with Tensorboard. All I'm doing is calling

writer = tf.summary.FileWriter('/tmp/tensorboard', sess.graph)

after sess.run(). Are there perhaps some "summaries" that I need to add to record memory usage?

Answer 1

Following the instruction here, you might actually have written all necessarry lines for memory and time to be displayed, but you also need to change the "Sessions runs" drop-down in the tensorboard GUI from default "None" to any of the options. Otherwise they are hidden. See graphic:

At least, this is what I needed to do to display them - the previous answers worked for me, but I needed to change the drop-down.

Answer 2

You need to add some RunOptions to your session run to your summary as explained in this document (Runtime statistics section).

Here is the piece of code which allows to do this:

merged = tf.summary.merge_all()
run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
run_metadata = tf.RunMetadata()
summary, _ = sess.run([merged, train_step],
                              feed_dict=...,
                              options=run_options,
                              run_metadata=run_metadata)
writer.add_run_metadata(run_metadata, 'step%d' % i)
writer.add_summary(summary, i)