How to keep calculated values in a Tensorflow graph (on the GPU)?

Question

How can we make sure that a calculated value will not be copied back to CPU/python memory, but is still available for calculations in the next step?

The following code obviously doesn't do it:

import tensorflow as tf

a = tf.Variable(tf.constant(1.),name="a")
b = tf.Variable(tf.constant(2.),name="b")
result = a + b
stored = result

with tf.Session() as s:
    val = s.run([result,stored],{a:1.,b:2.})
    print(val) # 3
    val=s.run([result],{a:4.,b:5.})
    print(val) # 9
    print(stored.eval()) # 3  NOPE:

Error : Attempting to use uninitialized value _recv_b_0

Answer 1

The answer is to store the value in a tf.Variable by storing to it using the assign operation:

working code:

import tensorflow as tf
with tf.Session() as s:
    a = tf.Variable(tf.constant(1.),name="a")
    b = tf.Variable(tf.constant(2.),name="b")
    result = a + b
    stored  = tf.Variable(tf.constant(0.),name="stored_sum")
    assign_op=stored.assign(result)
    val,_ = s.run([result,assign_op],{a:1.,b:2.})
    print(val) # 3
    val=s.run(result,{a:4.,b:5.})
    print(val[0]) # 9
    print(stored.eval()) # ok, still 3