C-extension in Python - return Py_BuildValue() memory leak problem

Question

I have a huge memory leak problem involving a C-extension I'm developing. In C, I have an array of doubles called A and an int variable called AnotherIntVariablethat I want to pass to Python. Well, in my C-extension module I do the following:

int i;  
PyObject *lst = PyList_New(len_A);  
PyObject *num;  
if(!lst)  
   return NULL;  
for(i=0;i<len_A;i++){  
   num=PyFloat_FromDouble(A[i]);  
   if(!num){  
      Py_DECREF(lst);  
      return NuLL;  
   }  
   PyList_SET_ITEM(lst,i,num);  
}  
free(A);  
return Py_BuildValue("Oi",lst,AnotherIntVariable)

So in Python i recieve this list and the int like this:

Pyt_A,Pyt_int=MyCModule.MyCFunction(...)

Where Pyt_A and Pyt_int are the list and the integer I get from my C-extension "MyCModule", from the function "MyCFunction" that I described earlier.

The problem is that, in Python, I use this Pyt_A array (so that's why I use Py_BuildValue instead of a simple return statement, to do an INCREF in order to save this variable for a moment from the garbage collector) but then I need to dereference it somehow in order to free that allocated memory. The problem is that I use the MyCFunction function several times, and this produces a memory leakage because I don't know how to dereference the array that I get in python in order to get rid of it.

I tried just returning the array by doing a return lst in the C part of the code instead of the Py_BuildValue("Oi",lst,AnotherIntVariable), but that only results in a Segmentation Fault when I try to use it in python (probably because the garbage collector did his work)...

...what am I missing here? Can anybody help me?

Answer 1

If you look at the documentation for Py_BuildValue (http://docs.python.org/3/c-api/arg.html#c.Py_BuildValue) you can see that under the O typecode, it says that the reference count of the passed in object is incremented by one (Note: an earlier section in that page describes the O typecode for PyArg_ParseTuple, which doesn't increment the reference count, but also isn't relevant here).

So, after the call to Py_BuildValue, the refcount for your list is 2, but you only want it to be 1.

Instead of returning the result of Py_BuildValue directly, save it to a PyObject pointer, decrement the lst reference count, then return your result.

You should be checking the result of the Py_BuildValue call anyway, since you also need to free num in the event that Py_BuildValue fails (i.e. returns NULL).

Answer 2

Thanks for clearing it up Ignacio, now it makes so much sense! Finally, the solution was to, instead of returning directly the Py_BuildValue, do:

free(A);  
PyObject *MyResult = Py_BuildValue("Oi",lst,AnotherIntVariable);  
Py_DECREF(lst);  
return MyResult

It worked like a charm!