Creating a Python package for a C extension-only module which is pre-built

Question

I want to create a package for a project that does not contain any .py source files, but is completely implemented as a Python C extension (resulting in an .so). Additionally, assume that the .so is already built by a separate build process (say CMake).

I know that setuptools/distutils minimally requires a directory structure:

• mymodule
  • __init__.py

But what I really want is for mymodule to be provided by a C extension (say mymodule.so) such that after installing the package, import mymodule has the same effect as directly importing mymodule.so.

I know that I could have this kind of directory structure:

• mymodule
  • __init__.py
  • mymodule_native.so

and have __init__.py be:

from mymodule_native import *

This kind of works, but an object A imported from mymodule will actually look like mymodule.mymodule_native.A.

Is there a more direct way?

Answer 1

It's possible if the extension is configured by setuptools.

For example:

from setuptools import setup, Extension

extension = Extension('mymodule', sources=[<..>])
setup('mymodule', ext_modules=[extension])

Once installed, the extension is available as import mymodule. Note that find_packages is not used.

This needs to be done by setuptools as it would otherwise require a packages setting if no ext_modules are provided.

However, this makes the .so module be installed directly under site-packages directory and will conflict with any non-extension python modules of the same name.

This is generally considered bad practice and most libraries use a bare python module with a single __init__.py, under which the extension is available.

You may in future add python code to your module for example and wish to separate the pure python code from extension code. Or you may want to add more than one extension which is not possible in this way, at least not while keeping the same package name.

So a structure like mymodule.<python modules> and mymodule.my_extension makes sense.

Personally I'd have separate name spaces for extension code vs python code and not do from <ext mod> import * in __init__.py.