All runs are not visible on TensorBoard

Question

I am using tensorboard to visualize three runs. I have a folder, logs, which contains three files:

2016-03-18_22-11-12
2016-03-18_22-11-27
2016-03-18_22-23-46

when I run tensorboard --logdir . (from logs), only 2016-03-18_22-23-46 is visible:

And if I delete 2016-03-18_22-23-46 from logs and restart tensorboard, then only 2016-03-18_22-11-27 is visible. Any idea of what's happening here?

Edit: the log files are (to my surprise), quite big: here is the result of du -h:

1,1G    ./2016-03-18_22-23-46
925M    ./2016-03-18_22-11-12
934M    ./2016-03-18_22-11-27
2,9G    .

EDIT: The above run structure can be obtained via logging and checkpointing into subdirectories of the tensorboard log-dir:

run_time = datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
FLAGS.train_dir = '/datalab/tf_runs/' + run_time #Directory to put the training data.
summary_writer = tf.train.SummaryWriter(FLAGS.train_dir, sess.graph_def)

Answer 1

Another solution is to use the --max_reload_threads option as follows:

tensorboard --logdir=runs --max_reload_threads 4

The no. of threads could be determined based on the fact that one thread can parse one run at a time.

Answer 2

Tensorboard takes a while to parse the log files. If you refresh the graph, you can see it showing more and more iterations. When it finished parsing one, the next run will pop up.

It is a bit annoying that there is no visual indication for this, maybe this is worth a feature request at https://github.com/tensorflow.