how to prepare cython submodules

Question

I have three questions, but related, and I not getting how can I split them well. I've found many information about those issues, like submodule extension, hierarchy, about an empty __init__.py file, and how to cythonize multiple pyx files. But when I try them together I cannot make them work.

I've prepared an small repo thinking in place together code samples of issues solved. I've check even the code of some of the projects listed that uses cython, but still not getting how to does three things at the same time.

• Empty __init__.py file:

In a project (where all the files are pyx (and pxd ifneedbe)), with a __init__.pyx that includes all of them, when there is a __init__.py file, then import it doesn't load the ".so" but the empty init.

• cythonize multiple files:

When instead of prepare a __init__.py that includes all the elements of a module. Like:

cythonize(['factory.pyx', 'version.pyx'])

the resulting ".so" import raises an exception:

>>> import barfoo
(...)
ImportError: dynamic module does not define init function (PyInit_barfoo)

It would be related with the previous question if it is necessary to write something in __init__.py.

• Submodule:

In fact, this is the main question. The singleton.pyx would part of a submodule, lets say utils to be used from other elements in the module.

For the sample there is a submodule (simply called subm) added in the setup.py as another extension. I've placed earlier than the main one (I don't know if this really does any difference, I didn't see it).

>>> import barfoo
>>> import barfoo.subm
(...)
ImportError: No module named 'barfoo.subm'

Separately, those recipes work, but together I cannot. The "__init__.py" seems to be necessary when there is a mix of "py" and "pyx" files. The examples explain how to cythonize with multiple files, but don't include the last key point for the import. And the submodules doesn't complete about how they can be imported from one place or another (import submodules when import the base one, or optional imports when they are specified).

Answer 1

Thanks to the comments from oz1 and DavidW, I've got the solution. Yes, those three things come together.

• Very important is the order when import in the setup.py. Even the PEP8 doesn't say that the imports should be alphabetically sorted, there are other guide lines (like reddit's) that does and I usually follow:

When import first cythonize and then setup, will cause that when cythonize(find_pyx()) is called, the result will be a list of distutils.extension.Extension objects.

from setuptools import setup, find_packages
from Cython.Build import cythonize

setuptools must be imported before cython and then the result of cythionize() will be a list of setuptools.extension.Extension objects that can be passed to the setup() call.

• Important to understand the meanings of the __init__'s:

All the __init__.pyx files with includes has been removed and each .pxy file produces its own .so binary.

The main module and the submodules will exist as long as their directories contain the __init__.py file like happen with a pure python code.

In the example I've linked, the file barfoo/__init__.py is not empty because I want that import barfoo provides access to elements like version() or Factory(). Then, this __init__.py is who imports them like a normal pure python.

• For the submodule:

Similar for the submodule and its own __init__.py file. In this example the import barfoo will do a from .factory import Factory, who will call from barfoo.subm import Bar, then the subm will be available.

But if the submodule is not imported in this secondary way, the user will have access to it with calls like import barfoo.subm.