How can I debug (potentially C-library related) memory issues using 64-bit Python on Windows?

Question

I have a python program that processes image frames with Python 2.7, PIL, OpenCV, and numpy/scipy. To the best of my knowledge, it does not maintain any lists of previous frame. Nevertheless, memory consumption increases steadily as the program processes more and more frames.

There are several good discussions of memory profiling solutions for Python, but they seem to focus on 32-bit or Linux solutions. What should I use with 64-bit Python 2.7 on Windows? Initial investigations suggest that the issue is with a C library. I am particularly interested in tools to help detect C library leaks or experience finding leaks in Python / OpenCV / PIL.

Answer 1

I've found the tools discussed here very helpful: http://mg.pov.lt/blog/hunting-python-memleaks.html

There is a version of his code here with some additions for measure numpy array sizes.

Answer 2

I had a similar sort of problem tracking down a severe memory leak in a numpy/scipy heavy code where none of the usual Python memory management tools and diagnostics detected the leak or hinted at its source.

In my case, the source of the leak was scipy interface code to the UMFPACK solver package, which was calling a C language initialization routine at every call of the interface object constructor, but never calling the de-initialization routine when the interface object was destroyed, resulting in scratch space and internal allocations leaking at the rate of about 15Mb a call. In an application with 10-20k calls, the impact was severe. Because the memory allocation was not done via the Python memory manager, things like heapy could not detect the leak.

I wound up having to use valgrind + "printf" style debugging to track down the culprit. You might need to look at non-python memory use analyzers and instrumentation tools to find out where the leak is coming from. I don't work in the Windows environment, and am not familiar with the standard toolchains, so I can't really suggest what to use. Perhaps someone else could chip in with some suggestions.