Does Tensorflow simplify a computational graph?

Question

I have a simple question and I was also searching already quiet a bit, but maybe I'm using the wrong keywords.

How does Tensorflow handle a given graph? If one has the simple graph:

x = tf.constant(1.0, name='input')

w = tf.constant0.8, name='weight')

b = tf.constant0.8, name='bias')

y_1 = tf.mul(w, x, name='output_1')

y_2 = tf.add(y_1, b, name='output_1')

The arithmetic statement is of course given by the computational graph, but is Tensorflow then kind of compiling and simplifying it in terms of saving time by not copying memories, etc.? So that it a 'condensed' version of the computational kernel is executed on the 'device' like CPU or GPU?

So that it reduces to something like that:

y_2 = tf.add(tf.mul(w, x), b, name='output_1')

Maybe somebody knows a good resource to learn more about how exactly Tensorflow runs under the hood without looking too deep into the source-code.

Thank you very much in advance!

Answer 1

TensorFlow includes various optimizations that can have the effect of simplifying a dataflow graph. In particular:

• TensorFlow will apply common subexpression elimination to avoid performing redundant computation. In the case of your example, this will not have much effect, but TensorFlow will observe that w and b are the same constant, and replace them with a single value.

• TensorFlow will apply constant propagation so that (computed) values that are the same in every execution of a subgraph will only be computed once. In your example, the entire expression is a constant, so TensorFlow will replace it with a single tf.constant() value corresponding to the result (1.6).

• If you use the experimental XLA compiler, TensorFlow will make more aggressive simplifications, and may be able to replace a subgraph with a single TensorFlow kernel, containing just-in-time compiled code. If in your example x were a tf.placeholder(), the remainder of the computation could be compiled into a single kernel with one input and one output.