Why should Py_INCREF(Py_None) be required before returning Py_None in C?

Question

Why should Py_INCREF(Py_None) be required before returning Py_None in C as follows?

Py_INCREF(Py_None);
return Py_None;

If Py_INCREF(Py_None) is omitted, what will happen?

Answer 1

Missing a Py_INCREF will result in an incorrect counting of references for Py_None, which may lead the interpreter to deallocate Py_None. Since Py_None is allocated statically in the Objects/object.c file:

PyObject _Py_NoneStruct = {
  _PyObject_EXTRA_INIT
  1, &PyNone_Type
};

And in Include/object.h there is the define:

#define Py_None (&_Py_NoneStruct)

So what will happen, is that the interpreter will crash with a fatal error:

Fatal Python error: deallocating None

Which is generated by the none_dealloc function in Objects/object.c:

/* ARGUSED */
static void
none_dealloc(PyObject* ignore)
{
    /* This should never get called, but we also don't want to SEGV if
     * we accidentally decref None out of existence.
     */
    Py_FatalError("deallocating None");
}

As stated by that comment, if NoneType didn't have its own deallocating function, you would obtain a Segmentation Fault, since a free call would be done on the stack.

You can test this copying the example in the tutorial, adding a call to Py_DECREF(Py_None) into the Noddy_name function, build the extension and do a loop calling that method.

---

In the general case a reference count of 0 can cause the program to fail in many different ways.

In particular python is free to re-use the memory used by the objects that were deallocated, which means that suddenly every reference to an object can become references to a random object(or to an empty memory location), and you could see things like:

>>> None   #or whatever object that was deallocated
<ARandomObjectYouNeverSawBefore object at ...>

(This actually happened to me sometimes, when writing C extensions. Some objects where turning into read only buffers at random times due to missing calls to Py_INCREF).

In other situations different kind of errors could be raised, or the interpreter could crash or segfault.

Answer 2

Py_None is really just another Python object, except without methods.

Python will count the references to any PyObject*. It doesn't matter if it is a string, integer, or None.

If you don't increment the reference count, the Python interpreter will eventually discard the object after its reference count hits 0, thinking that there aren't any pointers to the object. This means that the next time you try to do something with the return value, you will be following a pointer to a location in memory that is not guaranteed to hold Py_None (error, strange value, segmentation fault, etc.).

There are alternatives to having to remember to use Py_INCREF(Py_None):

return Py_BuildValue("");

or

Py_RETURN_NONE;