Tensorflow: var.eval() vs. var.initialized_value().eval()

Question

When I run the following Tensorflow commands I get different results. Anybody knows why?

import tensorflow as tf

sess = tf.Session()
var = tf.Variable(tf.truncated_normal([1,1], stddev=.1))
sess.run(tf.initialize_all_variables())

print var.eval(session=sess)
print var.initialized_value().eval(session=sess)

Produces:

[[-0.12024114]]
[[ 0.04141031]]

Answer 1

As you correctly worked out, the difference between evaluating a variable directly and evaluating var.initialized_value() is that evaluating var.initialized_value() will re-execute var.initializer, and—as a side effect—modify the value stored in var.

This leaves the question: why do we have initialized_value at all? The reason is that it helps when defining one variable in terms of another. For example, let's say we want to initialize two variables to the same random value. By default tf.global_variables_initializer does not specify the order in which variables are initialized. Therefore, if the initial value of a variable depends on another variable's value, it's likely that you'll get an error. The following code will not work reliably:

v1 = tf.Variable(tf.truncated_normal([20, 20]))
v2 = tf.Variable(v1)

init_op = tf.global_variables_initializer()
sess = tf.Session()
# The following line will non-deterministically crash with an error about
# using an uninitialized value.
sess.run(init_op)

Instead, you should define v2 in terms of v1.initialized_value(). This imposes an order on how the initializers may be executed, and ensures that v1 is initialized first:

v1 = tf.Variable(tf.truncated_normal([20, 20]))
v2 = tf.Variable(v1.initialized_value())

init_op = tf.global_variables_initializer()
sess = tf.Session()
# The following line will succeed.
sess.run(init_op)

Answer 2

Ok - easy. The reason for the different outputs is that the initialized_value() re-runs the initialization operation. Thus different results.

The initialized_value() method returns the value used to initialize the variable (in this case a random number).

I was looking to copy the values of a variable after some operations and ran into the initialized_value() method as a way to copy variables. However, this seems to only be true if you want to copy the initial values. It sounds kind of obvious from the name though...

import tensorflow as tf

sess = tf.Session()
var = tf.Variable([[1.234]])
sess.run(tf.initialize_all_variables())

print var.eval(session=sess)
print var.initialized_value().eval(session=sess)

var.assign_add([[2]]).eval(session=sess)

print var.eval(session=sess)
print var.initialized_value().eval(session=sess)

produces:

[[ 1.23399997]]
[[ 1.23399997]]

[[ 3.23399997]]
[[ 1.23399997]]