Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

What is the difference between variable_scope and name_scope? [duplicate]

What is the difference between variable_scope and name_scope? The variable scope tutorial talks about variable_scope implicitly opening name_scope. I also noticed that creating a variable in a name_scope automatically expands its name with the scope name as well. So, what is the difference?

like image 285
Andrzej Pronobis Avatar asked Dec 11 '15 03:12

Andrzej Pronobis


People also ask

What does TF Variable_scope do?

Variable scope allows you to create new variables and to share already created ones while providing checks to not create or share by accident. For details, see the Variable Scope How To, here we present only a few basic examples. The Variable Scope works as expected when the Eager Execution is Disabled.

What is TF Name_scope?

tf. name_scope( name, default_name=None, values=None ) This context manager validates that the given values are from the same graph, makes that graph the default graph, and pushes a name scope in that graph (see tf. Graph. name_scope for more details on that).

What is TF Get_variable?

The function tf. get_variable() returns the existing variable with the same name if it exists, and creates the variable with the specified shape and initializer if it does not exist.

What is reuse in Tensorflow?

reuse means sharing the same variable between different objects. If you want to share a variable, the second time you refer to that, you need to explicitly specify “reuse=True” in the variable scope of the variable that you wants to reuse, or. set the variable scope to “reuse=tf.AUTO_REUSE”


2 Answers

I had problems understanding the difference between variable_scope and name_scope (they looked almost the same) before I tried to visualize everything by creating a simple example:

import tensorflow as tf def scoping(fn, scope1, scope2, vals):     with fn(scope1):         a = tf.Variable(vals[0], name='a')         b = tf.get_variable('b', initializer=vals[1])         c = tf.constant(vals[2], name='c')         with fn(scope2):             d = tf.add(a * b, c, name='res')          print '\n  '.join([scope1, a.name, b.name, c.name, d.name]), '\n'     return d  d1 = scoping(tf.variable_scope, 'scope_vars', 'res', [1, 2, 3]) d2 = scoping(tf.name_scope,     'scope_name', 'res', [1, 2, 3])  with tf.Session() as sess:     writer = tf.summary.FileWriter('logs', sess.graph)     sess.run(tf.global_variables_initializer())     print sess.run([d1, d2])     writer.close() 

Here I create a function that creates some variables and constants and groups them in scopes (depending by the type I provided). In this function I also print the names of all the variables. After that I executes the graph to get values of the resulting values and save event-files to investigate them in tensorboard. If you run this, you will get the following:

scope_vars   scope_vars/a:0   scope_vars/b:0   scope_vars/c:0   scope_vars/res/res:0   scope_name   scope_name/a:0   b:0   scope_name/c:0   scope_name/res/res:0  

You see the similar pattern if you open TB (as you see b is outside of scope_name rectangular): enter image description here


This gives you the answer:

Now you see that tf.variable_scope() adds a prefix to the names of all variables (no matter how you create them), ops, constants. On the other hand tf.name_scope() ignores variables created with tf.get_variable() because it assumes that you know which variable and in which scope you wanted to use.

A good documentation on Sharing variables tells you that

tf.variable_scope(): Manages namespaces for names passed to tf.get_variable().

The same documentation provides a more details how does Variable Scope work and when it is useful.

like image 184
Salvador Dali Avatar answered Oct 08 '22 20:10

Salvador Dali


When you create a variable with tf.get_variable instead of tf.Variable, Tensorflow will start checking the names of the vars created with the same method to see if they collide. If they do, an exception will be raised. If you created a var with tf.get_variable and you try to change the prefix of your variable names by using the tf.name_scope context manager, this won't prevent the Tensorflow of raising an exception. Only tf.variable_scope context manager will effectively change the name of your var in this case. Or if you want to reuse the variable you should call scope.reuse_variables() before creating the var the second time.

In summary, tf.name_scope just add a prefix to all tensor created in that scope (except the vars created with tf.get_variable), and tf.variable_scope add a prefix to the variables created with tf.get_variable.

like image 39
cesarsalgado Avatar answered Oct 08 '22 21:10

cesarsalgado