I have a memory leak with TensorFlow. I refered to Tensorflow : Memory leak even while closing Session? to address my issue, and I followed the advices of the answer, that seemed to have solved the problem. However it does not work here.
In order to recreate the memory leak, I have created a simple example. First, I use this function (that I got here : How to get current CPU and RAM usage in Python?) to check the memory use of the python process :
def memory():
import os
import psutil
pid = os.getpid()
py = psutil.Process(pid)
memoryUse = py.memory_info()[0]/2.**30 # memory use in GB...I think
print('memory use:', memoryUse)
Then, everytime I call the build_model
function, the use of memory increases.
Here is the build_model
function that has a memory leak :
def build_model():
'''Model'''
tf.reset_default_graph()
with tf.Graph().as_default(), tf.Session() as sess:
tf.contrib.keras.backend.set_session(sess)
labels = tf.placeholder(tf.float32, shape=(None, 1))
input = tf.placeholder(tf.float32, shape=(None, 1))
x = tf.contrib.keras.layers.Dense(30, activation='relu', name='dense1')(input)
x1 = tf.contrib.keras.layers.Dropout(0.5)(x)
x2 = tf.contrib.keras.layers.Dense(30, activation='relu', name='dense2')(x1)
y = tf.contrib.keras.layers.Dense(1, activation='sigmoid', name='dense3')(x2)
loss = tf.reduce_mean(tf.contrib.keras.losses.binary_crossentropy(labels, y))
train_step = tf.train.AdamOptimizer(0.004).minimize(loss)
#Initialize all variables
init_op = tf.global_variables_initializer()
sess.run(init_op)
sess.close()
tf.reset_default_graph()
return
I would have thought that using the block with tf.Graph().as_default(), tf.Session() as sess:
and then closing the session and calling tf.reset_default_graph
would clear all the memory used by TensorFlow. Apparently it does not.
The memory leak can be recreated as following :
memory()
build_model()
memory()
build_model()
memory()
The output of this is (for my computer) :
memory use: 0.1794891357421875
memory use: 0.184417724609375
memory use: 0.18923568725585938
Clearly we can see that all the memory used by TensorFlow is not freed afterwards. Why?
I plotted the use of memory over 100 iterations of calling build_model
, and this is what I get :
I think that goes to show that there is a memory leak.
One of the leading causes of memory leaks across programming languages is the presence of unreferenced objects in the memory. Unreferenced objects are allocated in the memory but the reference to these objects has been deleted in the course of the control flow.
Memory leaks can still sneak up, even in the applications of a conscientious developer. There might still be situations where the application generates a substantial number of superfluous objects, thus depleting crucial memory resources, and sometimes resulting in the whole application's failure.
The problem was due to Tensorflow version 0.11. As of today Tensorflow 0.12 is out and the bug is resolved. Upgrade to a newer version and it should work as expected. Don't forget to call tf.contrib.keras.backend.clear_session()
at the end.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With