Organizing a package with Cython

Question

I have a package that requires Cython to build its extensions, and I am trying to tweak the setup.py file to streamline installation.

A simple

pip install git+git://<pkg-repo> 

throws an error

$ pip install git+https://<pkg-repo>
Downloading/unpacking git+https://<pkg-repo>
  Cloning https://<pkg-repo> to /tmp/pip-nFKHOM-build
  Running setup.py (path:/tmp/pip-nFKHOM-build/setup.py) egg_info for package from git+https://<pkg-repo>
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/tmp/pip-nFKHOM-build/setup.py", line 2, in <module>
        from Cython.Build import cythonize
    ImportError: No module named Cython.Build
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/tmp/pip-nFKHOM-build/setup.py", line 2, in <module>

    from Cython.Build import cythonize

ImportError: No module named Cython.Build

because of the Cython import before the Cython dependency is installed. This results in a multi-stage installation process:

pip install <deps> cython
pip install git+git://<pkg-repo>

which sucks. The relevant sections of setup.py are:

from setuptools import setup, find_packages
from Cython.Build import cythonize

setup(
    install_requires=[
        ...
        'cython>=0.19.1'
        ...
    ],
    ext_modules=cythonize([
        ...
        "pkg/io/const.pyx",
        ...
    ])
)

How can I change setup.py to still cythonize the ext_modules while relying on the install_requires to get Cython in the first place?

Answer 1

Since version 18.0, setuptools has support for Cython: you can just specify cython version in setup_requires and list needed sources in Extension, and setuptools will make sure to build them using Cython and install it if needed - there is no need to explicitly call cythonize() from your setup.py.

Your setup.py would look like this:

from setuptools import setup, Extension

setup(
    setup_requires=[
        ...
        'setuptools>=18.0',
        'cython>=0.19.1',
        ...
    ],
    ext_modules=Extension([
        ...
        "pkg/io/const.pyx",
        ...
    ])
)

I also did not know about this feature until I found it mentioned in this SO answer: https://stackoverflow.com/a/38057196/1509394.