Create package with cython so users can install it without having cython already installed

Question

I have a question. I would like to distribute my cython-powered packages, but I see no easy way to build them in setup.py. I would like setup.py to:

• most importantly: install my package without cython (from pre-generated C files or by installing cython beforehand)
• rebuild (run cythonize) package on sdist
• not need to hard-code list of my cython modules (just use glob or something)
• be able to work without .c files (should not be stored in git) or .pyx (might not be distributed). at least one of those sets will be always present of course.

Currently in my itchy package, I am using this quite complicated code:

import os
from glob import glob
from distutils.command.build_ext import build_ext as _build_ext
from distutils.command.sdist import sdist as _sdist
from distutils.core import setup
from distutils.core import Extension



def generate_extensions():
    return [
        # Compile cython-generated .c files into importable .so libraries.
        Extension(os.path.splitext(name)[0], [name])
        for name in C_FILES
    ]


# In distribution version, there are no pyx files, when you clone package from git, there will be no c files.
CYTHON_FILES = glob('itchy/*.pyx')
C_FILES = glob('itchy/*.c')
extensions = generate_extensions()


class build_ext(_build_ext):
    def run(self):
        # Compile cython files (.pyx, some of the .py) into .c files if Cython is available.
        try:
            from Cython.Build import cythonize
            if CYTHON_FILES:
                cythonize(CYTHON_FILES)

                # Update C_FILES in case they were originally missing.
                global C_FILES, extensions
                C_FILES = glob('itchy/*.c')
                extensions = generate_extensions()
            else:
                print('No .pyx files found, building extensions skipped. Pre-built versions will be used.')
        except ImportError:
            print('Cython is not installed, building extensions skipped. Pre-built versions will be used.')
            assert C_FILES, 'C files have to be present in distribution or Cython has to be installed'
        _build_ext.run(self)


class sdist(_sdist):
    def run(self):
        # Make sure the compiled Cython files in the distribution are up-to-date
        self.run_command("build_ext")
        _sdist.run(self)


setup(
    (...)
    ext_modules = extensions,
    cmdclass = {
        'build_ext': build_ext,
        'sdist': sdist,
    },
)

Answer 1

Usually done by attempting to import cython and adjusting extensions to either

1. Build pyx files with cython if cython is present
2. Build C files if cython is not present

For example:

try:
    from Cython.Distutils.extension import Extension
    from Cython.Distutils import build_ext
except ImportError:
    from setuptools import Extension
    USING_CYTHON = False
else:
    USING_CYTHON = True

ext = 'pyx' if USING_CYTHON else 'c'
sources = glob('my_module/*.%s' % (ext,))
extensions = [
    Extension(source.split('.')[0].replace(os.path.sep, '.'),
              sources=[source],
    )
for source in sources]
cmdclass = {'build_ext': build_ext} if USING_CYTHON else {}

setup(<..>, ext_modules=extensions, cmdclass=cmdclass)

The source.split stuff is needed as cythonized extension names need to be in the form my_module.ext while glob requires path names like my_module/ext.

See this repository for a real-world example.

You should, however, include .c files in your git repo as well as the distributable otherwise when it comes time to build a distribution the .c files will be re-built and may or may not be the same files as were built on your machine.

They may be built by another version of cython, for example, or on a different platform, producing different code.

Cython is a static compiler - it's recommended to commit the files it produces to repository.

It is strongly recommended that you distribute the generated .c files as well as your Cython sources, so that users can install your module without needing to have Cython available.

See Cython documentation on distributing modules.