Building Python submodules from C++ extensions via cmake

Question

I'm trying to incorporate a c++ extension as a submodule into an existing python library via cmake. Building the C++ extension works fine and importing it as a python module works, but not as the submodule of the header library.

I have the following directory structure:

frontend/
   foo.py
   bar.py
   backend/
      backend.cpp

The extension is bound to a python module via pybind:

PYBIND11_MODULE(backend, m)
{
    m.doc() = "backend c++ implementation"; // optional module docstring
    m.def("method", &method, "The method I want to call from python.");
}

In the CMakeLists.txt, the relevant line is:

pybind11_add_module(backend "frontend/backend/backend.cpp")

I've followed the instructions form here and here to write the setup.py script. I guess the most important lines look like this:

from setuptools import setup, Extension, find_packages
from setuptools.command.build_ext import build_ext
from setuptools.command.test import test as TestCommand

class CMakeExtension(Extension):
    def __init__(self, name, sourcedir=".", sources=[]):
        Extension.__init__(self, name, sources=[])


class CMakeBuild(build_ext):
    def run(self):
        build_directory = os.path.abspath(self.build_temp)
        if not os.path.exists(self.build_temp):
            os.makedirs(self.build_temp)

        cmake_list_dir = os.path.abspath(os.path.dirname(__file__))
        print("-" * 10, "Running CMake prepare", "-" * 40)
        subprocess.check_call(
            ["cmake", cmake_list_dir], cwd=self.build_temp,
        )

        print("-" * 10, "Building extensions", "-" * 40)
        cmake_cmd = ["cmake", "--build", "."] + self.build_args
        subprocess.check_call(cmake_cmd, cwd=self.build_temp)

        # Move from build temp to final position
        for ext in self.extensions:
            self.move_output(ext)

    def move_output(self, ext):
        build_temp = Path(self.build_temp).resolve()
        dest_path = Path(self.get_ext_fullpath(ext.name)).resolve()
        source_path = build_temp / self.get_ext_filename(ext.name)
        dest_directory = dest_path.parents[0]
        dest_directory.mkdir(parents=True, exist_ok=True)
        self.copy_file(source_path, dest_path)


extensions = [CMakeExtension("backend")]

setup(
    name="frontend",
    packages=["frontend"],
    ext_modules=extensions,
    cmdclass=dict(build_ext=CMakeBuild),
)

But this does not make backend a submodule of frontend, but instead a module on its own. So this works:

from backend import method

But to avoid naming issues with other libraries, what I would like to have is this:

from frontend.backend import method

Changing the naming in the pybinding or in the extension call to extensions = [CMakeExtension("frontend.backend")] does unfortunately not resolve my problem, the setup does not find the backend.<platform>.so shared library then, because it looks for frontend/backend.<platform>.so, which does not exist. How could I resolve this issue?

Answer 1

I think I've resolved the issue with the following lines:

Change in the setup.py file:

ext_modules = [
    Extension(
        "frontend.backend", sources=["frontend/backend/backend.cpp"]
    )
]

Change in the CMakeLists.txt file:

pybind11_add_module(backend "frontend/backend/backend.cpp")
set_target_properties( backend
    PROPERTIES
    ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/frontend"
    LIBRARY_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/frontend"
    RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/frontend"
)

The shared library object backend.platform.so must be located in the frontend directory. Neither the pybind module name nor the sourcefile .cpp should contain any "." in the names, because the get_ext_fullpath() method from build_ext will split by dots. Only the frontend directory containts an init.py file.