Python memory leaks trackdown?

Question

Somehow the memory my Python program takes more and more memory as it runs (the VIRT and RES) column of the "top" command keep increasing.

However, I double checked my code extremely carefully, and I am sure that there is no memory leaks (didn't use any dictionary, no global variables. It's just a main method calling a sub method for a number of times).

I used heapy to profile my memory usage by

from guppy import hpy;
heap = hpy();
.....
print heap.heap();

each time the main method calls the sub method. Surprisingly, it always gives the same output. But the memory usage just keeps growing.

I wonder if I didn't use heapy right, or VIRT and RES in "top" command do not really reflect the memory my code uses?

Or can anyone provide a better way to track down the memory usage in a Python script?

Thanks a lot!

Answer 1

Two possible cases:

• your function is pure Python, in which case possible causes include

  • you are storing an increasing number of large objects
  • you are having cycles of objects with a __del__ method, which the gc won't touch

  I'd suggest using the gc module and the gc.garbage and gc.get_objects function (see http://docs.python.org/library/gc.html#module-gc), to get list of existing objects, and you can then introspect them by looking at the __class__attribute of each object for instance to get information about the object's class.

• your function is at least partially written in C / C++, in which case the problem potentially is in that code. The advice above still applies, but won't be able to see all leaks: you will see leaks caused by missing calls to PY_DECREF, but not low level C/C++ allocations without a corresponding deallocation. For this you will need valgrind. See this question for more info on that topic