OpenCV Cython bridge leaking memory

Question

I have written an implementation of the VideoCapture class that works with Basler cameras. It is used like this:

import cv2
import PyBaslerCamera

video = PyBaslerCamera.PyBaslerCamera()
video.open(0)
while True:
    ret, image = video.read()
    cv2.imshow("Test", image)
    cv2.waitKey(1)

My Cython file looks like this:

# distutils: language = c++
# distutils: sources = BaslerCamera.cpp

from cython.operator cimport dereference as deref
from cpython.ref cimport PyObject
from libcpp cimport bool

cdef extern from "opencv2/core/core.hpp" namespace "cv":    
    cdef cppclass Mat:
        bool empty() const
        void release() const

    cdef cppclass _OutputArray:
        Mat getMat(int idx=-1) const



cdef extern from "cv2.cpp":
    void import_array()
    PyObject* pyopencv_from(const Mat&)
    int pyopencv_to(PyObject*, Mat&)

cdef Mat np2mat(object array):
    cdef Mat mat
    cdef PyObject* pyobject = <PyObject*> array
    pyopencv_to(pyobject, mat)
    return <Mat>mat

cdef object mat2np(const Mat &mat):
    return <object> pyopencv_from(mat)

cdef extern from "BaslerCamera.h" namespace "cv":
    cdef cppclass BaslerCamera:
        BaslerCamera()
        bool open(int index)
        bool isOpened()
        void release()
        bool grab()
        Mat retrieve()
        bool read(_OutputArray image)
        Mat read()
        bool set(int propId, double value)
        double get(int propId)
        BaslerCamera &operator>>(Mat &image)

cdef class PyBaslerCamera:
    cdef BaslerCamera *thisptr
    cdef Mat mat

    def __cinit__(self):
        print("PyBaslerCamera init")
        import_array()
        self.thisptr = new BaslerCamera()

    def __dealloc__(self):
        del self.thisptr

    def open(self, int index = 0):
        self.thisptr.open(index)

    def read(self):
        mat = self.thisptr.read()

        if mat.empty():
            return (False, None)
        else:
            out = mat2np(mat)
            return (True, out)

And I have used the cv2.cpp file from OpenCV: https://github.com/Itseez/opencv/blob/master/modules/python/src2/cv2.cpp

Now, everything works, I am getting the video stream from the camera, but the problem is that it leaks a lot (in a couple of seconds it will fill up my ram, which leads me to believe that it just leaks all the frames). Valgrind seems to confirm that

==21435== 1,050,624,000 bytes in 152 blocks are possibly lost in loss record 5,939 of 5,939
==21435==    at 0x4C2BBA0: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==21435==    by 0x20D7F3AB: ??? (in /usr/lib/python3/dist-packages/numpy/core/multiarray.cpython-34m-x86_64-linux-gnu.so)
==21435==    by 0x20D1BD89: ??? (in /usr/lib/python3/dist-packages/numpy/core/multiarray.cpython-34m-x86_64-linux-gnu.so)
==21435==    by 0x251D55E1: NumpyAllocator::allocate(int, int const*, int, void*, unsigned long*, int, cv::UMatUsageFlags) const (cv2.cpp:156)
==21435==    by 0xB983720: cv::Mat::create(int, int const*, int) (in /usr/local/lib/libopencv_core.so.3.0.0)
==21435==    by 0xB9B54C7: cv::_OutputArray::create(int, int, int, int, bool, int) const (in /usr/local/lib/libopencv_core.so.3.0.0)
==21435==    by 0xB810A7C: cv::Mat::copyTo(cv::_OutputArray const&) const (in /usr/local/lib/libopencv_core.so.3.0.0)
==21435==    by 0x251D44F9: pyopencv_from<cv::Mat> (cv2.cpp:211)
==21435==    by 0x251D44F9: __pyx_f_14PyBaslerCamera_mat2np (PyBaslerCamera.cpp:662)
==21435==    by 0x251D44F9: __pyx_pf_14PyBaslerCamera_14PyBaslerCamera_6read(__pyx_obj_14PyBaslerCamera_PyBaslerCamera*) [clone .isra.9] (PyBaslerCamera.cpp:973)
==21435==    by 0x503F5C: PyEval_EvalFrameEx (in /usr/bin/python3.4)
==21435==    by 0x5A9CB4: PyEval_EvalCodeEx (in /usr/bin/python3.4)
==21435==    by 0x5E7104: ??? (in /usr/bin/python3.4)
==21435==    by 0x5E71C8: PyRun_FileExFlags (in /usr/bin/python3.4)
==21435== 
==21435== LEAK SUMMARY:
==21435==    definitely lost: 165,107 bytes in 262 blocks
==21435==    indirectly lost: 179,724,840 bytes in 205 blocks
==21435==      possibly lost: 1,057,720,529 bytes in 646 blocks
==21435==    still reachable: 9,399,307 bytes in 10,288 blocks
==21435==         suppressed: 0 bytes in 0 blocks
==21435== Reachable blocks (those to which a pointer was found) are not shown.
==21435== To see them, rerun with: --leak-check=full --show-leak-kinds=all

It looks like the ndarrays created by the Numpy allocator don't get released, but I am at a loss at how to address this. Can anybody tell me how to properly release this memory?

Or if someone has a better suggestion on how to approach this whole cv::Mat to np array business I'm open to ideas.

Answer 1

The issue was that you needed to change the definition of pyopencv_from from PyObject* pyopencv_from(const Mat&) to object pyopencv_from(const Mat&):

# just illustrated in place
cdef extern from "cv2.cpp":
    void import_array()
    object pyopencv_from(const Mat&)
    # etc

# and a function that appears a bit later...
cdef object mat2np(const Mat &mat):
    # return <object> pyopencv_from(mat) # Problem line!
    # can now become:
    return pyopencv_from(mat)

This was based on a newsgroup post which references documentation that I think no longer exists. Quoted here:

whenver the Py_ function returns a new reference to a PyObject*, the return type is "object". When the function returns a borrowed reference, the return type is PyObject*. When Cython sees "object" as a return type it doesn't increment the reference count. When it sees PyObject* in order to use the result you must explicitly cast to <object>, and when you do that Cython increments the reference count wether you want it to or not, forcing you to an explicit DECREF (or leak memory). To avoid this we make the above convention.

With borrowed references if you do an explicit typecast to <object>, [Cython] generates an INCREF and DECREF so you have to be careful.

So the gist is:

• The object returned from pyopencv_from has a refcount of 1.

• If you tell Cython the function returns object it refcounts it appropriately (the generated code will probably show GOTREF which is a no-op except for debugging purposes followed by DECREF a bit later at which point the memory's freed).

• If you tell Cython the function returns PyObject* it does nothing because it just treats it as an arbitrary pointer type (OK - but you then have to do the refcounting)

• When you do the explicit case to <object> (see "problem line" in the listing above) it increments the reference by 1 (so it's now 2) to claim ownership, but only ever decrements it once. The reference count stays at 1 forever and the object is never freed.