Transfer a c++ object between cython and pybind11 using python capsules

Question

I have two c++ libraries that expose python API but using two different frameworks(pybind11 and cython). I need to transfer an object between them (both ways) using python capsules. Since cython and pybind11 use the python capsules different ways, is it even possible to make it work?

I have library A that defines a class Foo and exposes it with pybind11 to python. Library B exposes its API using cython. LibB owns a shared_ptr<Foo> which is a member of one of LibB's classes, say - Bar.

Bar returns the shared_ptr<Foo> member as a PyCapsule which I capture in pybind11 of the Foo class. I'm unpacking the shared_ptr<Foo> from the capsule, returning it to python and the user can operate on this object in python using pybind11 bindings for Foo.

Then I need to put it back in a capsule in pybind11 and return back to Bar.

Bar's python API operates on PyObject and PyCapsule because cython allows that. pybind11 and thus Foo's API does not accept those types and I'm forced to use pybind11::object and pybind11::capsule.

Everything works fine until the moment when I'm trying to use the pybind11::capsule created in pybind11, inside a cython method of class Bar which expects a PyCapsule*.

The shared_ptr<Foo> inside the pybind11::capsule is corrupted and my app crashes.

Has anyone tried to make those 2 libs talk to each other?

libA -> class Foo

namespace foo{
    class Foo {
    public:
        void foo() {...}
    }
}

libB -> class Bar

namespace bar {
    class Bar {
    public:
        PyObject* get_foo() {
            const char * capsule_name = "foo_in_capsule";
            return PyCapsule_New(&m_foo, capsule_name, nullptr);
        }

        static Bar fooToBar(PyObject * capsule) {
            void * foo_ptr = PyCapsule_GetPointer(capsule, "foo_in_capsule");
            auto foo  = static_cast<std::shared_ptr<foo::Foo>*>(foo_ptr);
            // here the shared_ptr is corrupted (garbage numbers returned for use_count() and get() )
            std::cout << "checking the capsule: " << foo->use_count() << " " << foo->get() << std::endl

            Bar b;
            b.m_foo = *foo; //this is what I would like to get
            return b;
        }

        std::shared_ptr<Foo> m_foo;
    };
}

pybind11 for Foo

void regclass_foo_Foo(py::module m)
{
    py::class_<foo::Foo, std::shared_ptr<foo::Foo>> foo(m, "Foo");
    foo.def("foo", &foo::Foo::foo);
    foo.def_static("from_capsule", [](py::object* capsule) {
        auto* pycapsule_ptr = capsule->ptr();
        auto* foo_ptr = reinterpret_cast<std::shared_ptr<foo::Foo>*>(PyCapsule_GetPointer(pycapsule_ptr, "foo_in_capsule"));
        return *foo_ptr;
    });
    foo.def_static("to_capsule", [](std::shared_ptr<foo::Foo>& foo_from_python) {
        auto pybind_capsule = py::capsule(&foo_from_python, "foo_in_capsule", nullptr);
        return pybind_capsule;
    });
}

cython for Bar

cdef extern from "bar.hpp" namespace "bar":
    cdef cppclass Bar:
        object get_foo() except +

def foo_to_bar(capsule):
    b = C.fooToBar(capsule)
    return b

putting it all together in python

from bar import Bar, foo_to_bar
from foo import Foo

bar = Bar(... some arguments ...)
capsule1 = bar.get_foo()

foo_from_capsule = Foo.from_capsule(capsule1)

// this is the important part - need to operate on foo using its python api
print("checking if foo works", foo_from_capsule.foo())
// and use it to create another bar object with a (possibly) modified foo object
capsule2 = Foo.to_capsule(foo_from_capsule)

bar2 = foo_to_bar(capsule2)

Answer 1

There's too many unfinished details in your code for me to even test your PyCapsule version. My view is that the issue is with the lifetime of the shared pointers - your capsule points to a shared pointer that's lifetime is tied to the Bar it's in. However, the capsule may outlive that. You should probably be creating a new shared_ptr<Foo>* (with new), pointing to that in your capsule, and defining a destructor (for the capsule) to delete it.

---

An outline of an alternative approach that I think should work better is as follows:

Write your classes purely in terms of C++ types, so get_foo and foo_to_bar just take/return shared_ptr<Foo>.

Define PyBar as a proper Cython class, rather than using capsules:

cdef public class PyBar [object PyBarStruct, type PyBarType]:
    cdef shared_ptr[Bar] ptr

cdef public PyBar PyBar_from_shared_ptr(shared_ptr[Bar] b):
    cdef PyBar x = PyBar()
    x.ptr = b
    return x

This generates a header file containing definitions of PyBarStruct and PyBarType (you probably don't need the latter). I also define a basic module-level function to create a PyBar from a shared pointer (and make that public as well, so it appears in the header too).

Then use PyBind11 to define a custom type-caster to/from shared_ptr<Bar>. load would be something like:

bool load(handle src, bool) {
        auto bar_mod = py::import("bar");
        auto bar_type = py::getattr(bar_mod,"Bar");
        if (!py::isinstance(src,bar_type)) {
            return false;
        }

        // now cast to my PyBarStruct
        auto ptr = reinterpret_cast<PyBarStruct*>(src.ptr());

        value  = ptr->ptr; // access the shared_ptr of the struct
    }

while the C++ to Python caster would be something like

 static handle cast(std::shared_ptr<Bar> src, return_value_policy /* policy */, handle /* parent */) {
     auto bar_mod = py::import("bar"); // See note...
     return PyBar_from_shared_ptr(src);
 }

I've ensured to include py::import("bar") in both functions because I don't think it's safe to use the Cython-defined functions until the module has been imported somewhere and importing it in the casters does ensure that.

This code is untested so almost certainly has errors, but should give a cleaner approach than PyCapsule.