How to use SWIG to wrap std::function objects?

Question

I have seen quite a few similar questions, but have not found a solution to my particular problem. I am attempting to SWIGify some C++11 code that uses std::function, so I can use it in my Java application.

I have encountered shared pointers like this:

virtual std::shared_ptr<some::ns::TheThing> getTheThing(unsigned short thingID);

and successfully handled them with the shared_ptr directive like so:

%shared_ptr(some::ns::TheThing);

I have encountered vectors of shared pointers like this:

virtual std::vector<std::shared_ptr<some::ns::TheThing>> getAllTheThings() const = 0;

and successfully handled them with a template like so:

%template(ThingVector) std::vector<std::shared_ptr<some::ns::TheThing>>;

Now I have a method like this:

 void registerThingCallback(std::function<void(std::shared_ptr<some::ns::TheThing>) > func);

and I cannot get SWIG to wrap it properly. I have tried using %callback, directors, %template, and %inline functional code, as I have seen examples with all of these things, but have not been able to get anything that seems close to working. Here is a little more context around the function call if that helps (sanitized and reduced):

thing_callback.h

#include <functional>

namespace some {
  namespace ns {

    /**
     * Hold some callbacks.
     */
    class ThingCallbacks {
    public:

        /**
         * Registers a callback 
         * @param func The callback function
         */
        void registerThingCallback(std::function<void(std::shared_ptr<some::ns::TheThing>) > func);

    };

  }
}

Update

Based on Flexo's great answer below, I am much closer to a solution. I was able to get the examples below working exactly as advertised. I tried incorporating it into my actual code, but ran into issues. To expand on my earlier simplified example, here is my definition of TheThing:

test_thing.h

#ifndef THE_THING_H
#define THE_THING_H

#include <string>

namespace some {
  namespace ns {

    class TheThing {
    public:

        virtual ~TheThing() {};

        virtual unsigned long longThing() const = 0;

        virtual std::string stringThing() const = 0;
    };
  }
}

#endif  /* THE_THING_H */

And here is my .i file. To have as few moving parts as possible, I've basically just taken the int and double from the code provided in the answer below, and replaced them with a shared pointer to my object.

func_thing_test.i

%module(directors="1") Thing
%include "stl.i"
%include "std_function.i"
%include "std_shared_ptr.i"

%shared_ptr(some::ns::TheThing);


%typemap(javadirectorin) std::shared_ptr<some::ns::TheThing> "new $typemap(jstype, some::ns::TheThing)($1,false)";
%typemap(directorin,descriptor="Lsome.ns.typemap(jstype, some::ns::TheThing);") std::shared_ptr<some::ns::TheThing> %{
     *($&1_type*)&j$1 = &$1;
%}


%include "test_thing.h"
%include "thing_callback.h"

%{
#include <memory>

#include "test_thing.h"
#include "thing_callback.h"

%}

%std_function(Functor, void, std::shared_ptr<some::ns::TheThing>);

%{
#include <iostream>
void add_and_print(std::shared_ptr<some::ns::TheThing> thing) {
  std::cout << "here\n";
}
%}

%callback("%s_cb");
void add_and_print(std::shared_ptr<some::ns::TheThing>);
%nocallback;

%inline %{
  std::function<void(std::shared_ptr<some::ns::TheThing>)> make_functor() {
    return [](std::shared_ptr<some::ns::TheThing>){
      std::cout << "make functor\n";
    };
  }

  void do_things(std::function<void(std::shared_ptr<some::ns::TheThing>)> in) {
      std::cout << "inside do things\n";
  }
%}

test_thing.h is what I just posted above, and thing_callback.h is the code I posted in my original question. This all compiles through the swig, c++, and Java chain without error, but it looks like swig is having a little trouble connecting the dots between the c++ and Java. It creates these three classes:

SWIGTYPE_p_f_std__function__f_std__shared_ptr__some__ns__TheThing____void____void
SWIGTYPE_p_f_std__shared_ptr__some__ns__TheThing____void
SWIGTYPE_p_std__functionT_void_fstd__shared_ptrT_some__ns__TheThing_tF_t

And unfortunately, most of the methods from the simple Java main code now take or return these objects, which make them fairly unusable. Any idea how to fix this? Thank you!

A little more detail for completeness: I am using the following three scripts to compile and run the code. The parameters are slightly different, but I don't think it matters. On my end it is set up as an Eclipse maven project. These scripts reside in the root of my project, headers and swig files reside in src/main/resources, java source files reside in src/main/java, and java compiled classes reside in target/classes. Eclipse performs the java compilation.

swigthing.sh

MODULE_NAME=Thing
PACKAGE=some.ns
OUTDIR=./src/main/java/some/ns
I_FILE=./src/main/resources/func_thing_test.i

mvn clean

rm $OUTDIR/*.*
mkdir -p $OUTDIR

swig -java -c++ -module $MODULE_NAME -package $PACKAGE -outdir $OUTDIR $I_FILE

./compileThingSwigTest.sh

compileThingSwigTest.sh

#!/bin/bash

pushd src/main/resources
g++ -c -std=gnu++11 -fpic \
func_thing_test_wrap.cxx \
-I/usr/lib/jvm/java/include \
-I/usr/lib/jvm/java/include/linux

g++ -shared func_thing_test_wrap.o -o libFunc.so
popd

runThingTest.sh

pushd target/classes
java -Xmx512M -Xms512M -Djava.library.path=. some.ns.test.RunThingTest
popd

Last Update

Fixed the code above to pass the right parameters to std_function. Now between the question and answer there is a complete working example of what I was after.

Answer 1

Although SWIG doesn't provide a std_function.i natively we can build one ourselves this with a bit of work. My answer here is a more generalised version of my a previous of mine answer, looking at this problem for a specific instance and targeting Python. I'll go through several iterations of it, which define a %std_function macro for generic std::function wrapping.

I'm assuming there are four things you want to achieve from a wrapping of std::function, which become our main requirements:

1. We want to be able to call std::function objects from within Java code.
2. The wrapped std::function objects need to get passed around like any other object, including crossing the language boundaries in either direction.
3. It should be possible to write std::function objects inside of Java, which can be passed back to C++ without having to modify existing C++ code that works on std::function objects (i.e. maintaining type erasure of std::function cross language)
4. We should be able to construct std::function objects in Java using C++ pointer to function types.

I'm going to work through these and show how we can achieve this. Where possible I'll keep the solution language agnostic too.

For the purposes of discussion I'm glossing over the shared_ptr part of your question, it doesn't actually change things because as you've got shared_ptr working that's actually sufficient to use it in this scenario too, it would just make my examples more verbose needlessly.

The solution I'm working towards is actually modelled after the existing shared_ptr support in SWIG. I've put together a test interface to illustrate how it will be used:

%module test
%include "std_function.i"

%std_function(Functor, void, int, double);

%{
#include <iostream>
%}

%inline %{
  std::function<void(int,double)> make_functor() {
    return [](int x, double y){
      std::cout << x << ", " << y << "\n";
    };
  }
%}

Basically to use this all you need do is include the file "std_function.i" and then use the macro %std_function which takes arguments as:

%std_function(Name, Ret, ...)

You call this once per instantiation of the std::function template you want to wrap, where Name is what you want to call the type in Java, Ret is the return type and then the remaining (variadic) arguments are the inputs to your function. So in my test interface above I'm basically looking to wrap std::function<void(int,double)>.

Writing a first version of "std_function.i" isn't actually too tricky. All you need to get basic working requirements #1 and #2 is:

%{
  #include <functional>
%}

%define %std_function(Name, Ret, ...)
%rename(Name) std::function<Ret(__VA_ARGS__)>;
%rename(call) std::function<Ret(__VA_ARGS__)>::operator();
namespace std {
  struct function<Ret(__VA_ARGS__)> {
    // Copy constructor
    function<Ret(__VA_ARGS__)>(const std::function<Ret(__VA_ARGS__)>&);

    // Call operator
    Ret operator()(__VA_ARGS__) const;
  };
}

%enddef

This includes the C++ header file in the generated wrapper code once and then sets up the macro for usage in interfaces. SWIG's support for C++11 variadic templates isn't actually very helpful for us in this usage scenario, so the macro I wrote basically re-implements the default template expansion functionality using C99 variadic macro arguments (which are much better supported). Coincidentally this means the SWIG code we're writing will work with 2.x or even some 1.3.x releases. (I tested with 2.x). Even if/when your version of SWIG does have %template support that works with std::function retaining this macro is still useful for the rest of the glue that makes it actually callable.

The manual expansion of the std:function template is limited to just the bits we care about for our usage: the actual operator() and a copy constructor that might come in handy.

The only other thing to be done is renaming operator() to something that matches the target language, e.g. for Java renaming it to be just a regular function called "call", or if you were targeting Python to __call__ or using tp_slots if required.

This is now sufficient to let our interface work, to demonstrate it I wrote a little bit of Java:

public class run {
    public static void main(String[] argv) {
        System.loadLibrary("test");
        test.make_functor().call(1,2.5);
    }
}

Which I compiled with:

swig2.0 -Wall -c++ -java  test.i
g++ -Wall -Wextra -std=c++11 test_wrap.cxx -o libtest.so -I/usr/lib/jvm/default-java/include/ -I/usr/lib/jvm/default-java/include/linux -shared -fPIC
javac run.java
LD_LIBRARY_PATH=. java run

and it worked.

---

Requirement #4 is pretty easy to cross off the list now at this point. All we need to do is tell SWIG there's another constructor in std::function which accepts compatible function pointers:

// Conversion constructor from function pointer
function<Ret(__VA_ARGS__)>(Ret(*const)(__VA_ARGS__));

And then we can use this with the %callback mechanism in SWIG, our test interface file becomes:

%module test
%include "std_function.i"

%std_function(Functor, void, int, double);

%{
#include <iostream>
void add_and_print(int a, double b) {
  std::cout << a+b << "\n";
}
%}

%callback("%s_cb");
void add_and_print(int a, double b);
%nocallback;

%inline %{
  std::function<void(int,double)> make_functor() {
    return [](int x, double y){
      std::cout << x << ", " << y << "\n";
    };
  }
%}

and the Java we use to call this is then:

public class run {
    public static void main(String[] argv) {
    System.loadLibrary("test");
    test.make_functor().call(1,2.5);
    new Functor(test.add_and_print_cb).call(3,4.5);
    }
}

Which we compile and run identically successfully at this point.

(Note that it's normal and desirable to see some Java classes created at this point that start with the name "SWIGTYPE_p_f_..." - these wrap the "pointer to function" types that are used by the pointer to function constructor and callback constants)

---

Requirement #3 is where things start to get trickier. Essentially we've hit the same problem as I answered on making SWIG generate an interface in Java previously, except now we're looking to do it within a macro more generically.

It turns out that in this instance because the interface we want to generate is fairly simple we can use some tricks inside our macro to make SWIG generate it for us.

The main thing that we need to do in order to make this work is to setup SWIG directors to provide cross-language polymorphism and allow something written in Java to implement a C++ interface. This is the class generated with the suffix "Impl" in my code.

To make things "feel right" to Java developers we want to still use the same type for both C++ and Java implemented std::function objects. Even if std::function::operator() were virtual we still don't want to have SWIG directors use that type directly as it's pretty common to pass std::function by value which would lead to type slicing problems. So when a Java developer extends our std::function objects and overrides call we need to do some extra work to make it so that C++ which uses that object actually calls the Java implementation given that we can't just use directors to handle this automatically.

So what we do looks a little bit weird. If you construct a Java object that is meant to implement std::function then there's a special, protected constructor for that. This constructor leaves the swigCPtr member variable, which normally points to a real C++ object as 0 and instead creates an anonymous wrapper object that implements the "Impl" interface and simply proxies everything back to the call member of the Java object.

We have another typemap too that gets applied, in Java, everywhere we pass a std::function object to C++. Its role is to detect which case we have - a C++ implemented std::function object, or a Java one. In the C++ case it does nothing special and everything proceeds as normal. In the Java case it takes the proxy object and asks C++ to convert it back to another, separate std::function instance which gets substituted instead.

This is enough to get us the behaviour we want in both languages without anything that feels weird on either side (other than a lot of mechanical lifting that happens transparently).

The catch here is that it's non-trivial to automatically construct the proxy object. Java has dynamic proxy classes as part of the reflection API, but these only implement interfaces, not extend abstract classes. One possibility I did try to use was void call(Object ...args) on the Java side, which is a variadic function argument. Whilst legal this didn't seem to actually override any cases in the super class as would be needed.

What I ended up doing was adapting some macros to iterate over variadic macro arguments in the way I wanted. This is a fairly sensible solution given that we already decided to use variadic C99 macro arguments for other reasons. This mechanism gets used four times in total in my solution, once in the function declaration and once in the delgated call for both Java and C++. (C++ has perfect forwarding properties retained and Java needs a typemap lookup to be performed, so they are distinct in each and every case).

There's also a custom typemap to simplify some of the Java code - in a void function it's not legal to write return other_void_function();, so we would need to special case void functions if it weren't for that.

So let's see what that looks like in reality. First up is the run.java I used for testing, it's only slightly modified from previous examples to add a Java implementation of the std::function object.

public class run extends Functor {
    public static void main(String[] argv) {
        System.loadLibrary("test");
        test.make_functor().call(1,2.5);

        new Functor(test.add_and_print_cb).call(3,4.5);

        Functor f = new run();
        test.do_things(f);
    }

    @Override
    public void call(int a, double b) {
        System.out.println("Java: " + a + ", " + b);
    }
}

The std_function.i is now substantially larger with all the changes outlined above:

%{
  #include <functional>
  #include <iostream>

  #ifndef SWIG_DIRECTORS
  #error "Directors must be enabled in your SWIG module for std_function.i to work correctly"
  #endif
%}

// These are the things we actually use
#define param(num,type) $typemap(jstype,type) arg ## num
#define unpack(num,type) arg##num
#define lvalref(num,type) type&& arg##num
#define forward(num,type) std::forward<type>(arg##num)

// This is the mechanics
#define FE_0(...)
#define FE_1(action,a1) action(0,a1)
#define FE_2(action,a1,a2) action(0,a1), action(1,a2)
#define FE_3(action,a1,a2,a3) action(0,a1), action(1,a2), action(2,a3)
#define FE_4(action,a1,a2,a3,a4) action(0,a1), action(1,a2), action(2,a3), action(3,a4)
#define FE_5(action,a1,a2,a3,a4,a5) action(0,a1), action(1,a2), action(2,a3), action(3,a4), action(4,a5)

#define GET_MACRO(_1,_2,_3,_4,_5,NAME,...) NAME
%define FOR_EACH(action,...) 
  GET_MACRO(__VA_ARGS__, FE_5, FE_4, FE_3, FE_2, FE_1, FE_0)(action,__VA_ARGS__)
%enddef

%define %std_function(Name, Ret, ...)

%feature("director") Name##Impl;
%typemap(javaclassmodifiers) Name##Impl "abstract class";

%{
  struct Name##Impl {
    virtual ~Name##Impl() {}
    virtual Ret call(__VA_ARGS__) = 0;
  };
%}

%javamethodmodifiers Name##Impl::call "abstract protected";
%typemap(javaout) Ret Name##Impl::call ";" // Suppress the body of the abstract method

struct Name##Impl {
  virtual ~Name##Impl();
protected:
  virtual Ret call(__VA_ARGS__) = 0;
};

%typemap(maybereturn) SWIGTYPE "return ";
%typemap(maybereturn) void "";

%typemap(javain) std::function<Ret(__VA_ARGS__)> "$javaclassname.getCPtr($javaclassname.makeNative($javainput))"
%typemap(javacode) std::function<Ret(__VA_ARGS__)> %{
  protected Name() {
    wrapper = new Name##Impl(){
      public $typemap(jstype, Ret) call(FOR_EACH(param, __VA_ARGS__)) {
    $typemap(maybereturn, Ret)Name.this.call(FOR_EACH(unpack, __VA_ARGS__));
      }
    };
    proxy = new $javaclassname(wrapper);
  }

  static $javaclassname makeNative($javaclassname in) {
    if (null == in.wrapper) return in;
    return in.proxy;
  }

  // Bot of these are retained to prevent garbage collection from happenign to early
  private Name##Impl wrapper;
  private $javaclassname proxy;
%}

%rename(Name) std::function<Ret(__VA_ARGS__)>;
%rename(call) std::function<Ret(__VA_ARGS__)>::operator();

namespace std {
  struct function<Ret(__VA_ARGS__)> {
    // Copy constructor
    function<Ret(__VA_ARGS__)>(const std::function<Ret(__VA_ARGS__)>&);

    // Call operator
    Ret operator()(__VA_ARGS__) const;

    // Conversion constructor from function pointer
    function<Ret(__VA_ARGS__)>(Ret(*const)(__VA_ARGS__));

    %extend {
      function<Ret(__VA_ARGS__)>(Name##Impl *in) {
    return new std::function<Ret(__VA_ARGS__)>([=](FOR_EACH(lvalref,__VA_ARGS__)){
          return in->call(FOR_EACH(forward,__VA_ARGS__));
    });
      }
    }
  };
}

%enddef

And test.i is slightly expanded to validate the Java -> C++ passing of std::function objects and enable directors:

%module(directors="1") test
%include "std_function.i"

%std_function(Functor, void, int, double);

%{
#include <iostream>
void add_and_print(int a, double b) {
  std::cout << a+b << "\n";
}
%}

%callback("%s_cb");
void add_and_print(int a, double b);
%nocallback;

%inline %{
  std::function<void(int,double)> make_functor() {
    return [](int x, double y){
      std::cout << x << ", " << y << "\n";
    };
  }

  void do_things(std::function<void(int,double)> in) {
    in(-1,666.6);
  }
%}

This compiled and ran as with the previous examples. It's worth noting that we've crossed into writing a lot of Java specific code - although the design would work for other languages if you were targeting Python it's a lot simpler to fix some of these issues using Python specific features.

There are two things I'd like to improve:

1. Use C++14 variadic lambdas to avoid the macro preprocessor magic I've used to keep them compatible with C++11. If you have C++ 14 the %extend constructor becomes:

   %extend {
     function<Ret(__VA_ARGS__)>(Name##Impl *in) {
       return new std::function<Ret(__VA_ARGS__)>([=](auto&& ...param){
         return in->call(std::forward<decltype(param)>(param)...);
       });
     }
   }
   

---

When it comes to using this macro with std::shared_ptr as expected the macro itself needs no changes. There is however a problem with the implementation of the javadirectorin and directorin typemaps that get applied, which do prevent things from "just working". This is true even with a build of SWIG from "trunk". (There's an outstanding question on combining directors and shared_ptr)

We can work around that though, by adding two additional typemaps in the main .i file of our module right after the call to %shared_ptr:

%shared_ptr(some::ns::TheThing);
%typemap(javadirectorin) std::shared_ptr<some::ns::TheThing> "new $typemap(jstype, some::ns::TheThing)($1,false)";
%typemap(directorin,descriptor="L$typemap(jstype, some::ns::TheThing);") std::shared_ptr<some::ns::TheThing> %{ 
  *($&1_type*)&j$1 = &$1;
%}

The first of these two typemaps is actually dead code because we forced the "call" method to be abstract in our abstract class, but it's easier to fix the compilation of this method than it is to suppress it. The second typemap is important. It's substantially similar to the normal "out" typemap in that it creates a jlong which is really just a representation of a C++ pointer, i.e. it prepares an object to go from C++, to Java.

Note that you might need to modify the descriptor attribute of the directorin typemap if you use packages in your module, either to "L$packagepath/$typemap(...);" or simply write it by hand.

This should remove the spurious "SWIGTYPE_p_sstd__shared_ptr..." type generated now as well. If you have virtual functions that return shared_ptr objects you'll need to write directorout and javadirectorout typemaps for them too. These can be base on the normal "in" typemap.

This was sufficient for my own simple testing with a modified Functor to work, at least with my version of SWIG checked out from the trunk today. (My test with 2.0.x failed and I didn't put much effort into making it work since this is a known work in progress area).