How to find leaks in Python ctypes libraries

Question

I'm working on a Python application which uses a number of open source third-party libraries. One of the libraries is based on ctypes, and I recently found more than 10 separate memory leaks in it. The causes of these leaks ranged from circular references on objects with explicit destructors (which Python can't garbage collect) to using c_char_p as a return type for functions returning non-const character arrays (resulting in the character arrays being converted automatically to Python strings and the original C-allocated arrays never being freed.)

I fixed the leaks I found and submitted a pull request to the author of the library. I've done some extremely informal testing by creating and deleting objects in a loop and watching Python's memory usage as I do so, and I think I've found all the leaks. However, as I'm planning to use this library in an application that I'd like to open source and hopefully have a few other people use, I'd like to be more sure than that. So my question is: is there a systematic way to find memory leaks in ctypes-based libraries?

During the process of fixing the leaks I've already found, I tried Heapy and objgraph but neither were particularly useful for this purpose. As far as I can tell, both of them will only show objects allocated on the Python heap, so they're of no use in finding leaks caused by improper handling of heap space allocated by C libraries. Is there a tool I can use in Python that can show me allocations on the C heap, and preferably also which Python objects, if any, refer to the allocated addresses?

Answer 1

You could try running the application under Valgrind. Valgrind's a useful tool for profiling memory use in compiled applications. This will at least detect the links and report their source.

You will certainly get false positives from Python calls. Check out this site for a nice description of how to use suppressions, which allow you to specifically ignore certain types of errors. See also Python's premade list of suppressions (here), and a description of why they are needed (here).