Python reference count and ctypes

Question

Hallo,

I have some troubles understanding the python reference count. What I want to do is return a tuple from c++ to python using the ctypes module.

C++:

PyObject* foo(...)
{

  ...
  return Py_BuildValue("(s, s)", value1, value2);
}

Python:

pointer = c_foo(...) # c_foo loaded with ctypes
obj = cast(pointer, py_object).value

I'm was not sure about the ref count of obj, so I tried sys.getrefcount() and got 3. I think it should be 2 (the getrefcount functions makes one ref itself).

Now I can't make Py_DECREF() before the return in C++ because the object gets deleted. Can I decrease the ref count in python?

edit What happens to the ref count when the cast function is called? I'm not really sure from the documentation below. http://docs.python.org/library/ctypes.html#ctypes.cast

ctypes.cast(obj, type) This function is similar to the cast operator in C. It returns a new instance of type which points to the same memory block as obj. type must be a pointer type, and obj must be an object that can be interpreted as a pointer.

Answer 1

On further research I found out that one can specify the return type of the function. http://docs.python.org/library/ctypes.html#callback-functions This makes the cast obsolete and the ref count is no longer a problem.

clib = ctypes.cdll.LoadLibrary('some.so')
c_foo = clib.c_foo
c_foo.restype = ctypes.py_object

As no additional answers were given I accept my new solution as the answer.

Answer 2

Your c++ code seems to be a classic wrapper using the official C-API and it's a bit weird since ctypes is usually used for using classic c types in python (like int, float, etc...).

I use personnally the C-API "alone" (without ctypes) but on my personnal experience, you don't have to worry about the reference counter in this case since you are returning a native python type with Py_BuildValue. When a function returns an object, the ownership of the returned object is given to the calling function.

You have to worry about Py_XINCREF/Py_XDECREF (better than Py_INCREF/Py_DECREF because it accepts NULL pointers) only when you want to change ownership of the object :

For example, you have created a wrapper of a map in python (let's call the typed object py_map). The element are of c++ class Foo and you have created an other python wrapper for them (let's call it py_Foo). If you create a function that wrap the [] operator, you are going to return a py_Foo object in python :

F = py_Map["key"]

but since the ownership is given to the calling function, you will call the destructor when you delete F and the map in c++ contains a pointer to a deallocated objet !

The solution is to write in c++ in the wrapper of [] :

...
PyObject* result; // My py_Foo object
Py_XINCREF(result); // transfer the ownership
return result;
}

You should take a look at the notion of borrowed and owned reference in python. This is essential to understand properly the reference counter.