TensorFlow supervisor prevents variable assignment: Graph is finalized and cannot be modified

Question

This code works fine:

import tensorflow as tf
x = tf.Variable(initial_value=0)

with tf.Session() as session:
    print session.run(x.assign(1))

But this code fails:

import tensorflow as tf
x = tf.Variable(initial_value=0)
supervisor = tf.train.Supervisor(logdir="/tmp")

with tf.Session() as session:
    print session.run(x.assign(1))

The only difference is the instantiation of a tf.train.Supervisor. Note that we don't even use the supervisor to create a managed session.

The error is:

python tf_supervisor_freeze.py
Traceback (most recent call last):
  File "tf_supervisor_freeze.py", line 6, in <module>
    print session.run(x.assign(1))
  File "<virtual_env_path>/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 522, in assign
    return state_ops.assign(self._variable, value, use_locking=use_locking)
  File "<virtual_env_path>/lib/python2.7/site-packages/tensorflow/python/ops/gen_state_ops.py", line 47, in assign
    use_locking=use_locking, name=name)
  File "<virtual_env_path>/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 491, in apply_op
    preferred_dtype=default_dtype)
  File "<virtual_env_path>/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 702, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "<virtual_env_path>/lib/python2.7/site-packages/tensorflow/python/framework/constant_op.py", line 110, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "<virtual_env_path>/lib/python2.7/site-packages/tensorflow/python/framework/constant_op.py", line 103, in constant
    attrs={"value": tensor_value, "dtype": dtype_value}, name=name).outputs[0]
  File "<virtual_env_path>/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2286, in create_op
    self._check_not_finalized()
  File "<virtual_env_path>/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2009, in _check_not_finalized
    raise RuntimeError("Graph is finalized and cannot be modified.")
RuntimeError: Graph is finalized and cannot be modified.

Process finished with exit code 1

Error remains if tf.train.Supervisor(logdir="/tmp", summary_op=None, saver=None) is used to disable some of the supervisor's services.

This problem was raised by somebody else on Github but no answer was provided there; the request was to raise the issue on StackOverflow instead. The only relevant StackOverflow question does not appear to address this specific case.

Answer 1

Just as the error says, you cannot modify the graph when the graph is finalized.

RuntimeError("Graph is finalized and cannot be modified.")

When execute this code tf.train.Supervisor(), the graph will be finalized. You can check code of it with path /tensorflow/python/training/supervisor.py in tensorflow of version 1.0.

And you will find this:

# The graph is not allowed to change anymore.
graph.finalize()

So you cannot modify the graph after Supervisor().