How to specify path to `.pxd` file in Cython

Question

My project has the following directory structure:

.
├── Makefile
├── pxd
├── pyx
│   ├── Landscaping.pyx
│   ├── Shrubbing.pxd
│   └── Shrubbing.pyx
└── setup.py

However, if I move Shrubbing.pxd anywhere else, say, into pxd/, I get the following error:

Error compiling Cython file:
------------------------------------------------------------
...
import pyx.Shrubbing
cimport Shrubbing
       ^
------------------------------------------------------------

pyx/Landscaping.pyx:2:8: 'Shrubbing.pxd' not found

Error compiling Cython file:
------------------------------------------------------------
...
import pyx.Shrubbing
cimport Shrubbing

cdef Shrubbing.Shrubbery sh
    ^
------------------------------------------------------------

This is strange because in setup.py I have:

from distutils.core import setup, Extension
from Cython.Build import cythonize
setup(ext_modules=cythonize([
    Extension(
        'pyx.Landscaping',
        sources=["pyx/Landscaping.pyx"],
        include_dirs=['pxd']), # <-- HERE
    Extension('pyx.Shrubbing', sources=["pyx/Shrubbing.pyx"])
]))

which clearly specifies the new directory for Shrubbing.pxd.

The source files are all very short, but to avoid cluttering this post, I will just post a link to a repository: https://github.com/lobachevzky/landscaping

Thanks for your help.

Answer 1

include_dirs is for C/C++ headers, not Cython pxd files.

In general it is best to keep related pyx/pxd files together in same directory, ie Shrubbing.pyx and Shrubbing.pxd should be in same directory.

To then use that from other modules, include a __init__.pxd and cimport via the name used in the Extension, eg pyx.Shrubbing as you would with Python modules.

If importing in python (not cimport), __init__.py should be included as well.

When using in the same module, OTOH, the .pxd needs to be available at runtime of that module, which means including it in Python search path.

If you want to organise the pxd files into separate dirs then link them symbolically within the module directory to make them available to the module, the dir containing an __init__.pxd or .py file.

It's a bit messy as Cython does not currently support relative imports, hence the need for linking when wanting to import from another dir.

See documentation for more details.

Answer 2

Strictly speaking this answer is broader than the specific question you are asking. It is more "how do you share pxd definitions between packages".

To sort this out for myself, I created a cython_example repo. It is not satisfying because it uses the old distutils tools rather than setuptools to prevent zip/egg from encapsulating the underlying files but may shed some light here. I'm actively looking for an improvement that will work with the automatic extraction of code from egg files so that setuptools can be used.

The core idea is to use the os.path.dirname(from_package.__file__) to use to add to the include path. See the setup.py file.

Update:

The cython email list provided the hint that using zip_unsafe when installing using setuptools would make everything work. See the updated working example and the update I added to the documentation .