What does tf.global_variables_initializer() do under the hood?

Question

There have been many cases (here and here) of TensorFlow users adding

init_op = tf.global_variables_initializer()

before defining any variables or operations and then getting an error along the lines of

Attempting to use uninitialized value

There is an explanation here but it makes no mention of the underlying tf.global_variables_initializer calls. It's almost copying TF API wholesale. This question focuses on the fact that there are still uninitialized values when some users call sess.run(init_op). Example code and an analysis into what tf.global_variables_initializer would be great.

Answer 1

The TensorFlow API has a very short section on tf.global_variables_initializer. It simply mentions:

This is just a shortcut for variable_initializer(global_variables()).

Tracing it to tf.variables_initializer, we can see that the usage of this function is as such:

tf.variables_initializer(var_list, name='init')

This means that we are implitcitly passing tf.global_variables as a var_list into tf.variables_initializer. If we have not defined any variables before calling tf.global_variables_initializer, var_list is essentially empty. The code below illustrates this:

import tensorflow as tf

with tf.Graph().as_default():
  # Nothing is printed
  for v in tf.global_variables():
    print v

  init_op = tf.global_variables_initializer()
  a = tf.Variable(0)
  b = tf.Variable(0)
  c = tf.Variable(0)

  # 3 Variables are printed here
  for v in tf.global_variables():
    print v

  with tf.Session() as sess:
    sess.run(init_op)
    print sess.run(a)

The 3 variables printed out are as such:

<tf.Variable 'Variable:0' shape=() dtype=int32_ref>
<tf.Variable 'Variable_1:0' shape=() dtype=int32_ref>
<tf.Variable 'Variable_2:0' shape=() dtype=int32_ref>

Running the above code as is would result in an error along the lines of what's mentioned above:

Attempting to use uninitialized value

Swapping the position of init_op to after a b c:

  a = tf.Variable(0)
  b = tf.Variable(0)
  c = tf.Variable(0)
  init_op = tf.global_variables_initializer()

would make it work.

Answer 2

The doc has actually a pretty detailed and useful section about variable initialization. You learn in particular that

Variable initializers must be run explicitly before other ops in your model can be run. The easiest way to do that is to add an op that runs all the variable initializers, and run that op before using the model. (..) Use tf.global_variables_initializer() to add an op to run variable initializers.

So basically tf.global_variables_initializer() go through all your variables, and call

sess.run(my_var.initializer)

tf.global_variables_initializer() does this automatically and at once. This function is a time-saver but technically you do not have to call it and could initialize your variables by other means (most frequent example: restoring weights from file).

How does it have knowledge of all variables? By default, when a Variable is created, it is added to tensorflow's GLOBAL_VARIABLES collection. So global_variables_initializer is basically iterating through the variables of this collection and calling their initializer.

However Variable let you choose collections to place it into. Should you exclude your variable from GLOBAL_VARIABLES, the created variable will not get initialized by tf_global_variables_initializer():

import tensorflow as tf

x = tf.Variable(0)
my_vars = []
y = tf.Variable(0, collections=my_vars)
sess = tf.InteractiveSession()
tf.global_variables_initializer().run() # does not initialize y!
print(x.eval())
# 0
print(y.eval())
# FailedPreconditionError: Attempting to use uninitialized value Variable_2