Boost.Python - Passing boost::python::object as argument to python function?

Question

So I'm working on a little project in which I'm using Python as an embedded scripting engine. So far I've not had much trouble with it using boost.python, but there's something I'd like to do with it if it's possible.

Basically, Python can be used to extend my C++ classes by adding functions and even data values to the class. I'd like to be able to have these persist in the C++ side, so one python function can add data members to a class, and then later the same instance passed to a different function will still have them. The goal here being to write a generic core engine in C++, and let users extend it in Python in any way they need without ever having to touch the C++.

So what I thought would work was that I would store a boost::python::object in the C++ class as a value self, and when calling the python from the C++, I'd send that python object through boost::python::ptr(), so that modifications on the python side would persist back to the C++ class. Unfortunately when I try this, I get the following error:

TypeError: No to_python (by-value) converter found for C++ type: boost::python::api::object

Is there any way of passing an object directly to a python function like that, or any other way I can go about this to achieve my desired result?

Thanks in advance for any help. :)

Answer 1

Got this fantastic solution from the c++sig mailing list.

Implement a std::map<std::string, boost::python::object> in the C++ class, then overload __getattr__() and __setattr__() to read from and write to that std::map. Then just send it to the python with boost::python::ptr() as usual, no need to keep an object around on the C++ side or send one to the python. It works perfectly.

Edit: I also found I had to override the __setattr__() function in a special way as it was breaking things I added with add_property(). Those things worked fine when getting them, since python checks a class's attributes before calling __getattr__(), but there's no such check with __setattr__(). It just calls it directly. So I had to make some changes to turn this into a full solution. Here's the full implementation of the solution:

First create a global variable:

boost::python::object PyMyModule_global;

Create a class as follows (with whatever other information you want to add to it):

class MyClass
{
public:
   //Python checks the class attributes before it calls __getattr__ so we don't have to do anything special here.
   boost::python::object Py_GetAttr(std::string str)
   {
      if(dict.find(str) == dict.end())
      {
         PyErr_SetString(PyExc_AttributeError, JFormat::format("MyClass instance has no attribute '{0}'", str).c_str());
         throw boost::python::error_already_set();
      }
      return dict[str];
   }

   //However, with __setattr__, python doesn't do anything with the class attributes first, it just calls __setattr__.
   //Which means anything that's been defined as a class attribute won't be modified here - including things set with
   //add_property(), def_readwrite(), etc.
   void Py_SetAttr(std::string str, boost::python::object val)
   {
      try
      {
         //First we check to see if the class has an attribute by this name.
         boost::python::object obj = PyMyModule_global["MyClass"].attr(str.c_str());
         //If so, we call the old cached __setattr__ function.
         PyMyModule_global["MyClass"].attr("__setattr_old__")(ptr(this), str, val);
      }
      catch(boost::python::error_already_set &e)
      {
         //If it threw an exception, that means that there is no such attribute.
         //Put it on the persistent dict.
         PyErr_Clear();
         dict[str] = val;
      }
   }
private:
   std::map<std::string, boost::python::object> dict;
};

Then define the python module as follows, adding whatever other defs and properties you want:

BOOST_PYTHON_MODULE(MyModule)
{
   boost::python::class_<MyClass>("MyClass", boost::python::no_init)
      .def("__getattr__", &MyClass::Py_GetAttr)
      .def("__setattr_new__", &MyClass::Py_SetAttr);
}

Then initialize python:

void PyInit()
{
   //Initialize module
   PyImport_AppendInittab( "MyModule", &initMyModule );
   //Initialize Python
   Py_Initialize();

   //Grab __main__ and its globals
   boost::python::object main = boost::python::import("__main__");
   boost::python::object global = main.attr("__dict__");

   //Import the module and grab its globals
   boost::python::object PyMyModule = boost::python::import("MyModule");
   global["MyModule"] = PyMyModule;
   PyMyModule_global = PyMyModule.attr("__dict__");

   //Overload MyClass's setattr, so that it will work with already defined attributes while persisting new ones
   PyMyModule_global["MyClass"].attr("__setattr_old__") = PyMyModule_global["MyClass"].attr("__setattr__");
   PyMyModule_global["MyClass"].attr("__setattr__") = PyMyModule_global["MyClass"].attr("__setattr_new__");
}

Once you've done all of this, you'll be able to persist changes to the instance made in python over to the C++. Anything that's defined in C++ as an attribute will be handled properly, and anything that's not will be appended to dict instead of the class's __dict__.