How to create nested namespace packages for setuptools distribution

Question

I'm developing a python project that will have separately distributable parts.

I have been able to accomplish part of my goal by making a namespace package. I have "sub1" and "sub2", both in namespace "lvl1". I can pip install these in development mode using "pip install -e" or python setup.py develop. I can import them with import lvl1.sub1 and import lvl1.sub2.

However, the project is massive and calls for nested namespaces. I want to import lvl1.lvl2.sub1 and import lvl1.lvl2.sub2. So both subpackages are in the same namespace ("lvl2"), which is itself in a namespace ("lvl1").

Desired conceptual structure:

lvl1/
    lvl2/
        sub1/
            code.py
            more_code.py
            ...
        sub2/
            code.py
            ...

Is there a way to do this and how?

Answer 1

Yes there is more than one way. Please read section "Nested namespace packages" in PEP 420.

In python >= 3.3, the easiest way to make nested namespace is to delete (do not include) file __init__.py in the specific folders ("lvl1" and "lvl2") in every distributable parts. In each of the setup.py, explicitly list all the packages in the deepest namespace.

"lvl1_part1/setup.py"

setup(
    name='lvl1_part1',
    ...
    zip_safe=False,
    packages=['lvl1.lvl2.sub1']
)

"lvl1_part2/setup.py"

setup(
    name='lvl1_part2',
    ...
    zip_safe=False,
    packages=['lvl1.lvl2.sub2']
)

The file structure for testing:

lvl1_part1/
           setup.py
           lvl1/
                lvl2/
                     sub1/
                          __init__.py
lvl1_part2/
           setup.py
           lvl1/
                lvl2/
                     sub2/
                          __init__.py

To make the above packages compatible to older python versions, please add the pkgutil magic file to each of the "lvl1" and "lvl2" folders.

Credits: The example above is modified from https://github.com/pypa/sample-namespace-packages/tree/master/pkgutil