About "tensorflow.initialize_all_variables()"

Question

I was wondering what're the differences between the following 2 pieces of code:

import tensorflow as tf
x = tf.Variable(0, name='x')
model = tf.initialize_all_variables()
with tf.Session() as session:
    for i in range(5):

        session.run(model)
        x = x + 1

        print(session.run(x))

---

import tensorflow as tf
x = tf.Variable(0, name='x')
model = tf.initialize_all_variables()
with tf.Session() as session:
    for i in range(5):

        x = x + 1
        session.run(model)

        print(session.run(x))

The only difference is the order of "x = x + 1" and "session.run(model)". I thought it would make a big difference in the output, as session.run(model) would initialize all variables. However, the two blocks of code output the same things...

The code is copied from a tutorial: http://learningtensorflow.com/lesson2/

Answer 1

Yeah, it's a bit tricky here. An important concept of Tensorflow is lazy evaluation, which means a Tensorflow graph of nodes is built first, and the evaluation of the graph only happens at session.run.

For this line of code x=x+1, the x here is of type Tensor, and the + here is the overloaded tf.add, so x=x+1 is actually building the graph, no computation happens yet; and at each iteration, the graph (a binary tree in this case) is added with one more layer (one more nested sum). session.run(model) will always initialize x to 0, session.run(x) will compute x based on the graph constructed so far in that iteration. For example in iteration 4, x is added by 1 for 4 times, because the graph at this iteration has 4 nested sums (or layers).

If this makes sense to you, I think "both code are essentially the same" will also make sense.

Note: strictly speaking, in the first iteration, the x in the right-hand side is of type Variable, but these are details, not the main point I'm trying to make...