TensorFlow's target pruning can't find nodes

Question

I wrote a Python script using the TensorFlow API, including a SummaryWriter that dumps the graph definition so I can look at it in TensorBoard.

When running the script, a NotFoundError is thrown saying PruneForTargets: Some target nodes not found: Reading/data_queue_EnqueueMany_1. As its name implies, the node in question was created by an enqueue_many call on a FIFOQueue (which is then started in a QueueRunner); it does in fact exist, and can be seen clearly in TensorBoard.

What could cause TensorFlow to not find some nodes?

Answer 1

This is a known issue that occurs when you start threads that access the TensorFlow graph (e.g. your QueueRunner) before adding more nodes to the graph. (The underlying tf.Graph data structure is not thread-safe for concurrent reads and writes.)

The solution is to move tf.train.start_queue_runners(sess) (and any other code that starts threads) after the last node is constructed. One way to double-check this is to add a call to tf.get_default_graph().finalize() immediately before calling start_queue_runners().