Should {tp_alloc, tp_dealloc} and {tp_new, tp_free} be considered as pairs?

Question

Is it true that whatever is created in tp_alloc should be destroyed in tp_dealloc? And similarly for {tp_new, tp_free}?

It looks like an obvious symmetry, but I would be grateful for clarification.

---

My actual use case is this: I have:

class OSClass : PyObject {...}

class Final : OSClass {...}

So the corresponding PyTypeObject pto has:

pto->tp_basicsize = sizeof(FinalClass)
pto->tp_dealloc = (destructor) 
                  [](PyObject* pyob) { PyMem_Free(pyob); };

However the new style class stores the PyObject and its corresponding C++ object separately from one another, and therefore works differently.

It creates the PyObject in tp_new, and the corresponding C++ object in tp_init.

And destroys both of them in tp_dealloc

Is this correct/optimal?

Code:

// extra void* to point to corresponding C++ object
pto->tp_basicsize = sizeof(PyObject) + sizeof(void*)

pto->tp_new = new_func;
pto->tp_init = init_func;
pto->tp_dealloc = dealloc_func;

static PyObject* new_func( PyTypeObject* subtype, PyObject* args, PyObject* kwds )
{
    // First we create the Python object.
    // The type-object's tp_basicsize is set to sizeof(Bridge)
    // (Note: We could maybe use PyType_GenericNew for this:
    //   http://stackoverflow.com/questions/573275/python-c-api-object-allocation )
    //
    PyObject* pyob = subtype->tp_alloc(subtype,0);

    Bridge* bridge = reinterpret_cast<Bridge*>(pyob);

    // We construct the C++ object later in init_func (below)
    bridge->m_pycxx_object = nullptr;

    return pyob;
}


static int init_func( PyObject* self, PyObject* args, PyObject* kwds )
{
    try
    {
        Object a = to_tuple(args);
        Object k = to_dict(kwds);

        Bridge* bridge{ reinterpret_cast<Bridge*>(self) };

        // NOTE: observe this is where we invoke the 
        //       constructor, but indirectly (i.e. through final)
        bridge->m_pycxx_object = new FinalClass{ bridge, a, k };
    }
    catch( Exception & )
    {
        return -1;
    }
    return 0;
}

static void dealloc_func( PyObject* pyob )
{
    auto final = static_cast<FinalClass*>( cxxbase_for(pyob) );

    delete final;
    PyMem_Free(pyob);

    COUT( "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" );
    //self->ob_type->tp_free(self);
}

Answer 1

From the tp_new documentation you have

The tp_new function should call subtype->tp_alloc(subtype, nitems) to allocate space for the object, and then do only as much further initialization as is absolutely necessary. Initialization that can safely be ignored or repeated should be placed in the tp_init handler. A good rule of thumb is that for immutable types, all initialization should take place in tp_new, while for mutable types, most initialization should be deferred to tp_init.

That's why you create the object itself in tp_new and initialise it in tp_init. Creating the C++ object is part of the initialisation. Since the tp_init documentation states

This function corresponds to the __init__() method of classes. Like __init__(), it is possible to create an instance without calling __init__(), and it is possible to reinitialize an instance by calling its __init__() method again.

You need to check for bridge->m_pycxx_object != nullptr and delete the already initialised instance on failure or raise an error.

In tp_dealloc you then destroy the Python object. Since the C++ object is part of this one, it needs to be destroyed there as well.

---

Back to the pairing: You call tp_alloc within tp_new and tp_free within tp_dealloc. So {tp_alloc, tp_free} and {tp_new, tp_dealloc} should be considered as pairs.