c++ back end call the python level defined callbacks with swig wrapper

Question

I am wrapping a library which was written in C++ to Python API libwebqq

There is a type which is defined in boost function .

typedef boost::function<void (std::string)> EventListener;

Python level can define "EventListener" variable callbacks.

There is also a map structure in C++ level which is event_map in class Adapter. The key type of event_map is a QQEvent enum type and the value type of event_map is class "Action" which wraps EvenListener.

class Action
{

    EventListener _callback;

    public:
    Action (){
        n_actions++;
    }

    Action(const EventListener & cb ){
    setCallback(cb);
    }

    virtual void operator()(std::string data) {
    _callback(data);
    }
    void setCallback(const EventListener & cb){
        _callback = cb;
    }

    virtual ~Action(){ std::cout<<"Destruct Action"<<std::endl; n_actions --; }
    static int n_actions;
};


class Adapter{

    std::map<QQEvent, Action > event_map;

public:
    Adapter();
    ~Adapter();
    void trigger( const QQEvent &event, const std::string data);
    void register_event_handler(QQEvent event, EventListener callback);
    bool is_event_registered(const QQEvent & event);
    void delete_event_handler(QQEvent event);
};

"register_event_handler" in class Adapter is the API to register a callback function to related event. And C++ back end will call it if event happened. But we need to implement the callbacks in python level. And I wrapped the callback type in "callback.i"

The problem is , when I call the register_event in test python script, a type error always occurs:

Traceback (most recent call last):
File "testwrapper.py", line 44, in <module>
worker = Worker()
File "testwrapper.py", line 28, in __init__
a.setCallback(self.on_message)
File "/home/devil/linux/libwebqq/wrappers/python/libwebqqpython.py", line 95, in setCallback
def setCallback(self, *args): return _libwebqqpython.Action_setCallback(self, *args)
TypeError: in method 'Action_setCallback', argument 2 of type 'EventListener const &'
Destruct Action

Please help to figure out the root cause of this type error and a solution to this problem.

Answer 1

The problem seems to be that you haven't included any code to map from a Python callable to your EventListener class. It's not provided for free, although it's something that comes up fairly regularly, e.g. here which acted as a reference for part of this answer.

Your question has quite a lot of code that's not really relevant to the problem and not quite complete either so I've produced a minimal header file to demonstrate the problem and solution with:

#include <boost/function.hpp>
#include <string>
#include <map>

typedef boost::function<void (std::string)> EventListener;

enum QQEvent { THING };

inline std::map<QQEvent, EventListener>& table() {
  static std::map<QQEvent, EventListener> map;
  return map;
}

inline const EventListener& register_handler(const QQEvent& e, const EventListener& l) {
  return table()[e] = l;
}

inline void test(const QQEvent& e)  {
  table()[e]("Testing");
}

Given that header file a simple wrapper would be:

%module test

%{
#include "test.hh"
%}

%include "test.hh"

I also put together a bit of Python to run this with:

import test

def f(x):
  print(x)

test.register_handler(test.THING, f)
test.test(test.THING)

With this I can reproduce the error you see:

LD_LIBRARY_PATH=. python3.1 run.py
Traceback (most recent call last):
  File "run.py", line 6, in 
    test.register_handler(test.THING, f)
TypeError: in method 'register_handler', argument 2 of type 'EventListener const &'

Hopefully we're on the same page now. There's a single version of register_handler that expects an object of type EventListener (SWIG's generated proxy for the type to be precise). We're not trying to pass an EventListener in when we call that function though - it's a Python Callable instead, with not much known on the C++ side - certainly not a type match or implicitly convertible. So we need to add some glue in our interface to mush the Python type into the real C++ type.

We do that by defining an entirely new type, which only exists internally to the SWIG wrapper code (i.e. within %{ }%). The type PyCallback has one purpose: hold a reference to the real Python thing we're working with and make it look/feel like a function in C++.

Once we've added that PyCallback implementation detail (which nobody gets to see) we then need to add another overload for register_handler, which takes a PyObject* directly and constructs the PyCallback+EventListener for us. Since this only exists for the purpose of wrapping we use %inline to declare, define and wrap all within the SWIG interface file. So our interface file now looks like:

%module test

%{
#include "test.hh"

class PyCallback
{
    PyObject *func;
    PyCallback& operator=(const PyCallback&); // Not allowed
public:
    PyCallback(const PyCallback& o) : func(o.func) {
      Py_XINCREF(func);
    }
    PyCallback(PyObject *func) : func(func) {
      Py_XINCREF(this->func);
      assert(PyCallable_Check(this->func));
    }
    ~PyCallback() {
      Py_XDECREF(func);
    }
    void operator()(const std::string& s) {
      if (!func || Py_None == func || !PyCallable_Check(func))
        return;
      PyObject *args = Py_BuildValue("(s)", s.c_str());
      PyObject *result = PyObject_Call(func,args,0);
      Py_DECREF(args);
      Py_XDECREF(result);
    }
};
%}

%include "test.hh"

%inline %{
  void register_handler(const QQEvent& e, PyObject *callback) {
    register_handler(e, PyCallback(callback));
  }
%}

At this point we now have enough for the original test Python to run successfully.

It's worth noting that we could have chosen to hide the original overload of register_handler, but in this instance I prefer not to - it's more of a feature than a mistake to leave that visible because it permits you to manipulate C++ defined callbacks also, e.g. you could get/set them from the Python side and expose some core ones as global variables.

In your actual code you'll want to do:

%extend Action {
  void setCallback(PyObject *callback) {
    $self->setCallback(PyCallback(callback));
  }
}

to overload Action::setCallback, after the line %include "test.hh", instead of the %inline I used in my example to overload the free function.

Finally you might want to expose operator() of your Action class using %rename, or you could chose to expose the callback_ member using the function pointer/member function trick.