What is the purpose of graph collections in TensorFlow?

Question

The API discusses Graph Collections which judging from the code are a general purpose key/data storage. What is the purpose of those collections?

Answer 1

Remember that under the hood, Tensorflow is a system for specifying and then executing computational data flow graphs. The graph collections are used as part of keeping track of the constructed graphs and how they must be executed. For example, when you create certain kinds of ops, such as tf.train.batch_join, the code that adds the op will also add some queue runners to the QUEUE_RUNNERS graph collection. Later, when you call start_queue_runners(), by default, it will look at the QUEUE_RUNNERS collection to know which runners to start.

Answer 2

I think there are at least two benefits for me so far:

1. when you distribute your program on multiple GPUs or machines it is convenient to collect losses from different devices which are in the same collection. Use the tf.add_n to add them to accumulate the loss.
2. To update a particular set of variables like weights and biases in my own way.

For instance:

import tensorflow as tf    
w = tf.Variable([1,2,3], collections=[tf.GraphKeys.WEIGHTS], dtype=tf.float32)    
w2 = tf.Variable([11,22,32], collections=[tf.GraphKeys.WEIGHTS], dtype=tf.float32)
weight_init_op = tf.variables_initializer(tf.get_collection_ref(tf.GraphKeys.WEIGHTS))
sess = tf.InteractiveSession()
weight_init_op.run()
for vari in tf.get_collection_ref(tf.GraphKeys.WEIGHTS): 
    tf.add_to_collection(tf.GraphKeys.UPDATE_OPS, vari.assign(0.2 * vari))
weight_update_ops = tf.get_collection_ref(tf.GraphKeys.UPDATE_OPS)
for op in weight_update_ops:
    print(op.eval())

The output:

[0.2 0.4 0.6]
[2.2 4.4 6.4]