Dynamically linking a shared library from a pybind11-wrapped code

Question

I am trying to add python bindings to a medium-sized C++ scientific code (some tens of thousands LOCs). I have managed to make it work without too many issues, but I have now incurred in an issue which I am incapable of solving myself. The code is organized as follows:

• All the classes and data structures are compiled in a library libcommon.a
• Executables are created by linking this library
• pybind11 is used to create a core.so python module

The bindings for the "main" parts work fine. Indeed, simulations launched from the standalone code or from python give the exact same results.

However, the code also supports a plugin-like system which can load shared libraries at runtime. These shared libraries contain classes that inherit from interfaces defined in the main code. It turns out that if I try to link these shared libraries from python I get the infamous "undefined symbol" errors. I have checked that these symbols are in the core.so module (using nm -D). In fact, simulations that perform the dynamic linking with the standalone code works perfectly (within the same folder and with the same input). Somehow, the shared lib cannot find the right symbols when called through python, but it has no issues when loaded by the standalone code. I am using CMake to build the system.

What follows is a MCE. Copy each file in a folder, copy (or link) the pybind11 folder in the same place and use the following commands:

mkdir build
cd build
cmake ..
make

which will generate a standalone binary and a python module. The standalone executable will produce the correct output. By contrast, using the following commands in python3 (that, at least in my head, should be equivalent) yields an error:

import core
b = core.load_plugin()

main.cpp

#include "Base.h"
#include "plugin_loader.h"

#include <iostream>

int main() {
    Base *d = load_plugin();
    if(d == NULL) {
        std::cerr << "No lib found" << std::endl;
        return 1;
    }
    d->foo();

    return 0;
}

Base.h

#ifndef BASE
#define BASE

struct Base {
    Base();
    virtual ~Base();

    virtual void foo();
};

#endif

Base.cpp

#include "Base.h"

#include <iostream>

Base::Base() {}

Base::~Base() {}

void Base::foo() {
    std::cout << "Hey, it's Base!" << std::endl;
}

plugin_loader.h

#ifndef LOADER
#define LOADER

#include "Base.h"

Base *load_plugin();

#endif

plugin_loader.cpp

#include "plugin_loader.h"

#include <dlfcn.h>
#include <iostream>

typedef Base* make_base();

Base *load_plugin() {
    void *handle = dlopen("./Derived.so", RTLD_LAZY | RTLD_GLOBAL);
    const char *dl_error = dlerror();
    if(dl_error != nullptr) {
        std::cerr << "Caught an error while opening shared library: " << dl_error << std::endl;
        return NULL;
    }
    make_base *entry = (make_base *) dlsym(handle, "make");

    return (Base *) entry();
}

Derived.h

#include "Base.h"

struct Derived : public Base {
    Derived();
    virtual ~Derived();
    void foo() override;
};

extern "C" Base *make() {
    return new Derived();
}

Derived.cpp

#include "Derived.h"

#include <iostream>

Derived::Derived() {}

Derived::~Derived() {}

void Derived::foo() {
    std::cout << "Hey, it's Derived!" << std::endl;
}

bindings.cpp

#include <pybind11/pybind11.h>

#include "Base.h"
#include "plugin_loader.h"

PYBIND11_MODULE(core, m) {
        pybind11::class_<Base, std::shared_ptr<Base>> base(m, "Base");

        base.def(pybind11::init<>());
        base.def("foo", &Base::foo);

        m.def("load_plugin", &load_plugin);
}

CMakeLists.txt

PROJECT(foobar)

# compile the library
ADD_LIBRARY(common SHARED Base.cpp plugin_loader.cpp)
TARGET_LINK_LIBRARIES(common ${CMAKE_DL_LIBS})
SET_TARGET_PROPERTIES(common PROPERTIES POSITION_INDEPENDENT_CODE ON)

# compile the standalone code
ADD_EXECUTABLE(standalone main.cpp)
TARGET_LINK_LIBRARIES(standalone common)

# compile the "plugin"
SET(CMAKE_SHARED_LIBRARY_PREFIX "")
ADD_LIBRARY(Derived SHARED Derived.cpp)

# compile the bindings
ADD_SUBDIRECTORY(pybind11)
INCLUDE_DIRECTORIES( ${PROJECT_SOURCE_DIR}/pybind11/include )

FIND_PACKAGE( PythonLibs 3 REQUIRED )
INCLUDE_DIRECTORIES( ${PYTHON_INCLUDE_DIRS} )

ADD_LIBRARY(_oxpy_lib STATIC bindings.cpp)
TARGET_LINK_LIBRARIES(_oxpy_lib ${PYTHON_LIBRARIES} common)
SET_TARGET_PROPERTIES(_oxpy_lib PROPERTIES POSITION_INDEPENDENT_CODE ON)

pybind11_add_module(core SHARED bindings.cpp)
TARGET_LINK_LIBRARIES(core PRIVATE _oxpy_lib)

Answer 1

You are right, symbols from imported library are not visible because core loaded without RTLD_GLOBAL flag set. You can fix that with a couple of extra lines on python side:

import sys, os
sys.setdlopenflags(os.RTLD_GLOBAL | os.RTLD_LAZY)

import core
b = core.load_plugin()

From sys.setdlopenflags() doc:

To share symbols across extension modules, call as sys.setdlopenflags(os.RTLD_GLOBAL). Symbolic names for the flag values can be found in the os module (RTLD_xxx constants, e.g. os.RTLD_LAZY).