How to pass a C++ object to another C++ object with Boost.Python

Question

I have some C++ code that defines two classes, A and B. B takes an instance of A during construction. I have wrapped A with Boost.Python so that Python can create instances of A, as well as subclasses. I want to do the same with B.

class A {
    public:
        A(long n, long x, long y) : _n(n), _x(x), _y(y) {};
        long get_n() { return _n; }
        long get_x() { return _x; }
        long get_y() { return _y; }
    private:
        long _n, _x, _y;
};

class B {
    public:
        B(A a) : _a(a) {};
        doSomething() { ... };
    private:
        A _a;
};

While wrapping B, I needed to work out how to pass an instance of A to B's constructor. I did some digging and the solution I found was to write a "converter" class:

struct A_from_python_A {
    static void * convertible(PyObject* obj_ptr) {
        // assume it is, for now...
        return obj_ptr;
    }

    // Convert obj_ptr into an A instance
    static void construct(PyObject* obj_ptr,
                      boost::python::converter::rvalue_from_python_stage1_data* data) {
        // extract 'n':
        PyObject * n_ptr = PyObject_CallMethod(obj_ptr, (char*)"get_n", (char*)"()");
        long n_val = 0;
        if (n_ptr == NULL) {
            cout << "... an exception occurred (get_n) ..." << endl;
        } else {
            n_val = PyInt_AsLong(n_ptr);
            Py_DECREF(n_ptr);
        }

        // [snip] - also do the same for x, y

        // Grab pointer to memory into which to construct the new A
        void* storage = (
            (boost::python::converter::rvalue_from_python_storage<A>*)
            data)->storage.bytes;

        // in-place construct the new A using the data
        // extracted from the python object
        new (storage) A(n_val, x_val, y_val);

        // Stash the memory chunk pointer for later use by boost.python
        data->convertible = storage;
    }

    // register converter functions
    A_from_python_A() {
        boost::python::converter::registry::push_back(
            &convertible,
            &construct,
            boost::python::type_id<A>());
    }
};

Then I register this with:

BOOST_PYTHON_MODULE(interpolation_ext)
{
    // register the from-python converter for A
    A_from_python_A();

    class_<A>("A", init<long, long, long>())
        ;

    class_<B>("B", init<object>())
        ;
}

Convertible and construct are methods that answer the "is this convertible?" and "how to convert?" questions respectively. I have observed that the construct() method is non-trivial - it has to reach into A's PyObject*, extract all relevant fields, then rebuild a C++ instance that it then passes to B's constructor. Because A contains some private fields, it has to do this via public access mechanisms (whereas with a pure Python object it wouldn't have to, right?). This seems to work.

However, is the field extraction in the 'construct' function really necessary? It seems laborious. If A is a compound object, it could get very complicated, and possibly require one converter to invoke another. I perhaps understand the requirement if A is a Python class, but if the A instance originated from the C++ side, is there a way to determine that this is the case, and then simply get a handle (e.g. pointer) to this 'native' object, as a shortcut?

Here's the associated python code:

from my_ext import A, B
a = A(1,2,3)
b = B(a)
b.doSomething()

Answer 1

In short, define B's wrapper as:

class_<B>( "B", init< A >() )

instead of

class_<B>( "B", init< object >() )

When defining a wrapper for class in Boost.Python (at least in 1.50), the class_ template generates convert and construct functions. This allows A to be converted to and constructed from a A's wrapper. These PyObject conversions have strict type-checking, and require that the following be true in python: isinstance( obj, A ).

Custom converters are often used to support:

• Automatic conversions to and from existing Python types. For example, converting std::pair< long, long > to and from a PyTupleObject.
• Duck-typing. For example, having B accept class D, which is not derived from A, as long as D provides a compatible interface.

---

Constructing B from an instance of A

Since A and B are neither existing Python types nor is duck-typing required, custom converters are not necessary. For B to take an instance of A, it can be as simple as specifying that init takes an A.

Here is a simplified example of A and B, where B can be constructed from an A.

class A
{
public:
  A( long n ) : n_( n ) {};
  long n() { return n_; }
private:
  long n_;
};

class B
{
public:
  B( A a ) : a_( a ) {};
  long doSomething() { return a_.n() * 2; }
private:
  A a_;
};

And the wrappers would be defined as:

using namespace boost::python;
BOOST_PYTHON_MODULE(example)
{
  class_< A >( "A", init< long >() )
    ;

  class_<B>( "B", init< A >() )
    .def( "doSomething", &B::doSomething )
    ;
}

B's wrapper explicitly indicates that it will be constructed from an A object via init< A >(). Also, A's interface is not fully exposed to the Python objects, as no wrapper was defined for the A::n() function.

>>> from example import A, B
>>> a = A( 1 )
>>> b = B( a )
>>> b.doSomething()
2

This also works for types that are derived from A. For example:

>>> from example import A, B
>>> class C( A ):
...     def __init__( self, n ):
...         A.__init__( self, n )
... 
>>> c = C( 2 )
>>> b = B( c )
>>> b.doSomething()
4

However, duck-typing is not enabled.

>>> from example import A, B
>>> class E: pass
... 
>>> e = E()
>>> b = B( e )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
Boost.Python.ArgumentError: Python argument types in
    B.__init__(B, instance)
did not match C++ signature:
    __init__(_object*, A)

---

Constructing B from an object that is convertible to A.

To support the case where B can be constructed from an object that provides a compatible interface, then custom converters are required. Although wrappers were not previously generated for A::n(), lets continue with the statement that an object can be converted to A if the object provides a get_num() method that returns an int.

First, write an A_from_python struct that provides converter and constructors functions.

struct A_from_python
{
  static void* convertible( PyObject* obj_ptr )
  {
    // assume it is, for now...
    return obj_ptr;
  }

  // Convert obj_ptr into an A instance
  static void construct(
    PyObject* obj_ptr,
    boost::python::converter::rvalue_from_python_stage1_data* data)
  {
    std::cout << "constructing A from ";
    PyObject_Print( obj_ptr, stdout, 0 );
    std::cout << std::endl;

    // Obtain a handle to the 'get_num' method on the python object.
    // If it does not exists, then throw.
    PyObject* n_ptr = 
      boost::python::expect_non_null( 
        PyObject_CallMethod( obj_ptr,
                             (char*)"get_num",
                             (char*)"()"  ));

    long n_val = 0;
    n_val = PyInt_AsLong( n_ptr );
    Py_DECREF( n_ptr );

    // Grab pointer to memory into which to construct the new A
    void* storage = (
      (boost::python::converter::rvalue_from_python_storage< A >*)
       data)->storage.bytes;

    // in-place construct the new A using the data
    // extracted from the python object
    new ( storage ) A( n_val );

    // Stash the memory chunk pointer for later use by boost.python
    data->convertible = storage;
  }

  A_from_python()
  {
    boost::python::converter::registry::push_back(
      &convertible,
      &construct,
      boost::python::type_id< A >() );
  }
};

boost::python::expect_non_null is used to throw an exception if NULL is returned. This helps provide the duck-typing guarantee that the python object must provide a get_num method. If the PyObject is known to be an instance of given type, then it is possible to use boost::python::api::handle and boost::python::api::object to directly extract the type, and avoid having to generically make calls through the PyObject interface.

Next, register the converter in the module.

using namespace boost::python;
BOOST_PYTHON_MODULE(example)
{
  // register the from-python converter for A
  A_from_python();

  class_< A >( "A", init< long >() )
    ;

  class_<B>( "B", init< A >() )
    .def( "doSomething", &B::doSomething )
    ;
}

No changes have occurred to A, B, or their associated wrapper definitions. The auto-conversion functions were created, and then defined/registered within the module.

>>> from example import A, B
>>> a = A( 4 )
>>> b = B( a )
>>> b.doSomething()
8
>>> class D:
...     def __init__( self, n ):
...         self.n = n
...     def get_num( self ):
...         return self.n
... 
>>> d = D( 5 )
>>> b = B( d )
constructing A from <__main__.D instance at 0xb7f7340c>
>>> b.doSomething()
10
>>> class E: pass
...
>>> e = E()
>>> b = B( e )
constructing A from <__main__.E instance at 0xb7f7520c>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: get_num

D::get_num() exists, and thus A is constructed from an instance of D when D is passed to B's constructor. However, E::get_num() does not exists, and an exception is raised when trying to construct A from an instance of E.

---

An alternative conversion solution.

Implementing duck-typing via the C-API can become very complicated with larger types. An alternative solution is perform the duck-typing in python, and distribute the python file alongside the library.

example_ext.py will import the A and B types, as well as monkey patch B's constructor:

from example import A, B

def monkey_patch_B():
    # Store handle to original init provided by Boost.
    original_init = B.__init__

    # Construct an A object via duck-typing.
    def construct_A( obj ):
        return A( obj.get_num() )

    # Create a new init that will delegate to the original init.
    def new_init( self, obj ):
        # If obj is an instance of A, use it.  Otherwise, construct
        # an instance of A from object.
        a = obj if isinstance( obj, A ) else construct_A ( obj )

        # Delegate to the original init.
        return original_init( self, a )

    # Rebind the new_init.
    B.__init__ = new_init

monkey_patch_B()

The only change required to the end-user is to import example_ext instead of example:

>>> from example_ext import A, B
>>> a = A( 6 )
>>> b = B( a )
>>> b.doSomething()
12
>>> class D:
...     def __init__( self, n ):
...         self.n = n
...     def get_num( self ):
...         return self.n
... 
>>> d = D( 7 )
>>> b = B( d )
>>> b.doSomething()
14
>>> class E: pass
... 
>>> e = E()
>>> b = B( e )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "example_ext.py", line 15, in new_init
    a = obj if isinstance( obj, A ) else construct_A ( obj )
  File "example_ext.py", line 9, in construct_A
    return A( obj.get_num() )
AttributeError: E instance has no attribute 'get_num'

Since the patched constructor guarantees an instance of A will be passed to B, the A_from_python::construct will not get invoked. Hence, the missing print statements in the output.

While this approach avoids the C-API, making it easier to perform duck-typing, it does have one major trade-off in that it requires parts of the API to be specifically patched for conversions. On the other hand, no patching is required when auto-type conversion functions are available.

---

Also, for what it is worth, access control in both C++ and Python is intended to protect against accidental misuse. Neither protect against deliberately obtaining access to members with private visibility. It is much easier to do in Python, but it is specifically permitted in the C++ Standard through explicit template instantiations.