Working with multiple graphs in TensorFlow

Question

Can someone explain to me how name_scope works in TensorFlow?

Suppose I have the following code:

import tensorflow as tf

g1 = tf.Graph()
with g1.as_default() as g:
    with g.name_scope( "g1" ) as scope:
        matrix1 = tf.constant([[3., 3.]])
        matrix2 = tf.constant([[2.],[2.]])
        product = tf.matmul(matrix1, matrix2)

tf.reset_default_graph()

g2 = tf.Graph()
with g2.as_default() as g:
    with g.name_scope( "g2" ) as scope:
        matrix1 = tf.constant([[4., 4.]])
        matrix2 = tf.constant([[5.],[5.]])
        product = tf.matmul(matrix1, matrix2)

tf.reset_default_graph()

with tf.Session( graph = g1 ) as sess:
    result = sess.run( product )
    print( result )

When I run this code I get the following error message:

Tensor Tensor("g2/MatMul:0", shape=(1, 1), dtype=float32) is not an element of this graph.

I agree "g2/MatMul" is not an element of graph g1, but why is it selecting "g2/MatMul" when the session graph is set to g1? Why doesn't it select "g1/MatMul"?

---

Edit

The following code seems to work:

import tensorflow as tf

g1 = tf.Graph()
with g1.as_default() as g:
    with g.name_scope( "g1" ) as g1_scope:
        matrix1 = tf.constant([[3., 3.]])
        matrix2 = tf.constant([[2.],[2.]])
        product = tf.matmul( matrix1, matrix2, name = "product")

tf.reset_default_graph()

g2 = tf.Graph()
with g2.as_default() as g:
    with g.name_scope( "g2" ) as g2_scope:
        matrix1 = tf.constant([[4., 4.]])
        matrix2 = tf.constant([[5.],[5.]])
        product = tf.matmul( matrix1, matrix2, name = "product" )

tf.reset_default_graph()

use_g1 = False

if ( use_g1 ):
    g = g1
    scope = g1_scope
else:
    g = g2
    scope = g2_scope

with tf.Session( graph = g ) as sess:
    tf.initialize_all_variables()
    result = sess.run( sess.graph.get_tensor_by_name( scope + "product:0" ) )
    print( result )

By flipping the switch use_g1, graph g1 or g2 will run in the session. Is this the way name scoping was meant to work?

Answer 1

Your product is a global variable, and you've set it to point to "g2/MatMul".

In particular

Try

print product

and you'll see

Tensor("g2/MatMul:0", shape=(1, 1), dtype=float32)

So the system takes "g2/MatMul:0" since that's the Tensor's name, and tries to find it in the graph g1 since that's the graph you set for the session. Incidentally you can see all nodes in the graph print [n.name for n in g1.as_graph_def().node]

Generally, using more than one graph is rarely useful. You can't merge them and can't pass tensors between them. I'd recommend just doing

tf.reset_default_graph()
a = tf.Constant(2)
sess = tf.InteractiveSession()
....

This way you'll have one default graph and one default session and you can omit specifying graph or session in most cases. If you ever need to refer to them explicitly, you can get them from tf.get_default_graph() or tf.get_default_session()

Answer 2

This is quite the necro, but it's still a top search result for questions related to this and I thought it might help to make very explicit something that the prior answers (which are correct) note in passing:

The Q's variable product is a python variable. As such, it points to an object: When defined, it points to the tf.Tensor output of the matmul tf.Operation defined in the name_scope 'g1'. It is later redefined to point to a different object, the tf.Tensor output of 'g2'. This python variable has never heard of tf.name_scopes and doesn't care.

That is why you need to do lookups by the name attributes of the tensorflow objects... Those, by use of name_scopes, are unique and accessible. Or generate distinct python variables--that are unique and accessible according to python scoping rules--to point to each tf.Tensor object you want to reference.

Dunno if this is helpful for anybody else, but if I ever forget, I'll thank my past self for this.