Install header-only library with Python

Question

I have a header-only C++ library that I use in my Python extensions. I would like to be able to install them to Python's include path, such that I can compile extensions very easily with python3 setup.py build. I'm partly able, but there are two things that I cannot get working (see below):

1. How can I use python3 setup.py install to install the header files? Currently I only get some *.egg file, but not the headers installed.

2. How can I retain the module's file structure? Currently the file structure is erroneously flattened.

What works

With the following setup.py

from setuptools import setup

setup(
   name        = 'so',
   description = 'Example',
   headers     = [
      'so.h',
   ],
)

I can upload the module to PyPi:

python3 setup.py bdist_wheel --universal
twine upload dist/*

and then install it using pip:

pip3 install so

On my system then I then find the header here

/usr/local/include/python3.6m/so/so.h

which is available when I compile the extensions with Python.

How can I use 'python3 setup.py install'?

Using this strategy I cannot simply run

python3 setup.py install

In that case some so*.egg is installed, but the headers are not stored somewhere where they are available to the compiler.

How to retain a file structure?

When the module is a bit more complicated, and there is some directory hierarchy I also run to problems. For the following setup.py

from setuptools import setup

setup(
  name        = 'so',
  description = 'Example',
  headers     = [
    'so.h',
    'so/implementation.h',
  ],
)

The problem is that the headers are installed to

/usr/local/include/python3.6m/so/so.h
/usr/local/include/python3.6m/so/implementation.h

thus flattening the original file structure.

How can I fix both issues?

Answer 1

How can I use python3 setup.py install to install the header files?

Unfortunately, you can't as long as you're using setuptools. What happens under the hood when you call setuptools.setup()? An egg installer is being built (bdist_egg command) and installed (via easy_install), and neither bdist_egg nor easy_install support including/installing headers. Although the distribution object carries the headers info, it is never requested during the install command. This is an old well-known problem that was never resolved because apparently, installation of header files doesn't fit into the egg build/install procedure.

You thus have three options (or at least three options I know of). Two of them (both inducing a switch to distutils) are not recommended and are provided for completeness sake only:

Bare distutils install (not recommended)

$ sed 's/from setuptools import setup/from distutils.core import setup/' setup.py

This way, the good ol' distutils will take care of the installation when doing python setup.py install, no egg installer will be built and install_headers will be invoked. However, this also includes giving up all the features of setuptools including additional keyword args in setup() and all the other good stuff, needless to say that packages installed via distutils can't be uninstalled with pip.

old-and-unmanageable install (not recommended)

Run the installation with

$ python setup.py install --old-and-unmanageable

This is a switch setuptools provides if you explicitly wish to run the distutils install. The egg installer is not built, instead, the distutils.command.install.install is invoked. Thus, the installation is the same as with bare distutils install.

Drawbacks of this approach: same as with bare distutils install plus: setuptools condemns usage of the switch; if you forget to provide it, you end with installing eggs and have to redo the installation.

Replace python setup.py install with pip install (recommended)

pip is capable of installing packages from source directories; just issue

$ pip install dir/

assuming dir contains the setup.py. This way, a wheel file is built from sources (same as in bdist_wheel; actually, this command is being run first) and installed, managing the installation of header files just fine.

How can I retain the module's file structure?

You will have to tweak the install_headers command a bit:

import os
from distutils.command.install_headers import install_headers as install_headers_orig
from setuptools import setup

class install_headers(install_headers_orig):

    def run(self):
        headers = self.distribution.headers or []
        for header in headers:
            dst = os.path.join(self.install_dir, os.path.dirname(header))
            self.mkpath(dst)
            (out, _) = self.copy_file(header, dst)
            self.outfiles.append(out)

setup(
    name='so',
    headers=['h1.h', 'subtree/h2.h'],
    cmdclass={'install_headers': install_headers},
    ...
)

What is essential here is the line

dst = os.path.join(self.install_dir, os.path.dirname(header))

The vanilla install_headers copies the header files directly to install_dir; the above line in the overloaded install_headers command additionally takes care of the eventual subdirectories in header filenames. When installing the package, the subdirectories should be retained now:

$ pip show -f so | grep include
  ../../../include/site/python3.6/so/h1.h
  ../../../include/site/python3.6/so/subtree/h2.h