Is it possible to split a SWIG module for compilation, but rejoin it when linking?

Question

I hit this issue about two years ago when I first implemented our SWIG bindings. As soon as we exposed a large amount of code we got to the point where SWIG would output C++ files so large the compiler could not handle them. The only way I could get around the issue was to split up the interfaces into multiple modules and to compile them separately.

This has several downsides:

• Each module must know about dependencies in other modules. I have a script to generate the interface files which handles this side of things, but it adds extra complexity.

• Each additional module increases the time that the dynamic linker requires to load in the code. I have added an init.py file that imports all the submodules, so that the fact that the code is split up is transparent to the user, but what is always visible is the long load times.

I'm currently reviewing our build scripts / build process and I wanted to see if I could find a solution to this issue that was better than what I have now. Ideally, I'd have one shared library containing all the wrapper code.

Does anyone know how I can acheive this with SWIG? I've seen some custom code written in Ruby for a specific project, where the output is post-processed to make this possible, but when I looked at the feasibility for Python wrappers it does not look so easy.

Answer 1

I just did equivalent hack for TCL library: I use several SWIG modules, generating several .cpp files that are compiled in several .o files but compile them all in a single .so file that is loaded by a single TCL "load" command.

The idea is to creates a top swig module (Top) that calls initialization functions of all sub-modules (Sub1 and Sub2):

%module Top
%header %{
  extern "C" {
    SWIGEXPORT int Sub1_Init(Tcl_Interp *);
    SWIGEXPORT int Sub2_Init(Tcl_Interp *);
  }
%}
%init %{
    if (Sub1_Init(interp) != TCL_OK) {return TCL_ERROR;}
    if (Sub2_Init(interp) != TCL_OK) {return TCL_ERROR;}
%}

There's nothing special in the submodules files. I end up with file Top.so that I load from TCL with command "load ./Top.so"

I don't know python but's likely to be similar. You may need to understand how the python extensions are loaded, though.