What is the difference between variable_scope
and name_scope
? The variable scope tutorial talks about variable_scope
implicitly opening name_scope
. I also noticed that creating a variable in a name_scope
automatically expands its name with the scope name as well. So, what is the difference?
Variable scope allows you to create new variables and to share already created ones while providing checks to not create or share by accident. For details, see the Variable Scope How To, here we present only a few basic examples. The Variable Scope works as expected when the Eager Execution is Disabled.
tf. name_scope( name, default_name=None, values=None ) This context manager validates that the given values are from the same graph, makes that graph the default graph, and pushes a name scope in that graph (see tf. Graph. name_scope for more details on that).
The function tf. get_variable() returns the existing variable with the same name if it exists, and creates the variable with the specified shape and initializer if it does not exist.
reuse means sharing the same variable between different objects. If you want to share a variable, the second time you refer to that, you need to explicitly specify “reuse=True” in the variable scope of the variable that you wants to reuse, or. set the variable scope to “reuse=tf.AUTO_REUSE”
I had problems understanding the difference between variable_scope and name_scope (they looked almost the same) before I tried to visualize everything by creating a simple example:
import tensorflow as tf def scoping(fn, scope1, scope2, vals): with fn(scope1): a = tf.Variable(vals[0], name='a') b = tf.get_variable('b', initializer=vals[1]) c = tf.constant(vals[2], name='c') with fn(scope2): d = tf.add(a * b, c, name='res') print '\n '.join([scope1, a.name, b.name, c.name, d.name]), '\n' return d d1 = scoping(tf.variable_scope, 'scope_vars', 'res', [1, 2, 3]) d2 = scoping(tf.name_scope, 'scope_name', 'res', [1, 2, 3]) with tf.Session() as sess: writer = tf.summary.FileWriter('logs', sess.graph) sess.run(tf.global_variables_initializer()) print sess.run([d1, d2]) writer.close()
Here I create a function that creates some variables and constants and groups them in scopes (depending by the type I provided). In this function I also print the names of all the variables. After that I executes the graph to get values of the resulting values and save event-files to investigate them in tensorboard. If you run this, you will get the following:
scope_vars scope_vars/a:0 scope_vars/b:0 scope_vars/c:0 scope_vars/res/res:0 scope_name scope_name/a:0 b:0 scope_name/c:0 scope_name/res/res:0
You see the similar pattern if you open TB (as you see b
is outside of scope_name
rectangular):
This gives you the answer:
Now you see that tf.variable_scope()
adds a prefix to the names of all variables (no matter how you create them), ops, constants. On the other hand tf.name_scope()
ignores variables created with tf.get_variable()
because it assumes that you know which variable and in which scope you wanted to use.
A good documentation on Sharing variables tells you that
tf.variable_scope()
: Manages namespaces for names passed totf.get_variable()
.
The same documentation provides a more details how does Variable Scope work and when it is useful.
When you create a variable with tf.get_variable
instead of tf.Variable
, Tensorflow will start checking the names of the vars created with the same method to see if they collide. If they do, an exception will be raised. If you created a var with tf.get_variable
and you try to change the prefix of your variable names by using the tf.name_scope
context manager, this won't prevent the Tensorflow of raising an exception. Only tf.variable_scope
context manager will effectively change the name of your var in this case. Or if you want to reuse the variable you should call scope.reuse_variables() before creating the var the second time.
In summary, tf.name_scope
just add a prefix to all tensor created in that scope (except the vars created with tf.get_variable
), and tf.variable_scope
add a prefix to the variables created with tf.get_variable
.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With