How can python libraries of different projects be in the same package?

Question

Let's say you have a big (python) software project called Ninja. There are several parts of the project like a server and a client, but also a common infrastructure library, which contains common classes and tools. Naturally I would create a package structure like this: ninja.core, ninja.server and ninja.client, where both the server and the client import ninja.core in some way. For development purposes (we use eclipse with Subversion) the core, server and client are maintained in different projects, thus resulting in a folder structure like this:

eclipse_workspace
|
>-Ninja_core
| |
| >-ninja
|   |
|   >-core
|
>-Ninja_client
| |
| >-ninja
|   |
|   >-client
.
.
.

Having a java background I assumed this would be possible, but it turned out (read: import errors) that this does not work. As pointed out in this answer, it is generally not possible to have both ninja.core and ninja.client unless they are subpackages of the same package ninja, which they are not. This gives rise to:

Approach A:

• Put the entire code in a single eclipse/svn project and have only one package ninja with the according subpackages.

In a production environment we want to be able to install core and server but not client or core and client but not server. I might be mistaken, but as far as I understand python packages, this would not be possible with approach A. Keeping the projects separate but compatible, it appears to be useful to use packages with the names ninja_core,ninja_client and ninja_server, which does in fact solve the import problems and has everything running smoothly in the development setup. To meet the requirement of being able to install server and client independently, I came up with the idea of:

Approach B:

• Create a new project called ninja which contains the package ninja
• let the __init__.py of ninja import the other libraries (if installed) in a way that they appear to be inside ninja.

I have not gotten this to work so far and I think it might not even be possible. I was thinking of something like this:

# __init__.py of ninja
import ninja_core as core
# or this:
from ninja_core import core
# or this:
import ninja_core.core

I tried these and have gotten import errors again. After googling for ways to combine python packages and not finding anything related to my problem I came here.

I was thinking that maybe the whole thing is a design fault. Should client and server even be in the same package if they can be installed independently? Is it a bad idea to want to be able to install the client and server independently? Why can I extend packages in java, but not in python? What is the idea behind that?

tl;dr

I am developing the Ninja library where a user should be able to do import ninja.client and import ninja.server. It needs to be possible to install the libraries for client and server separately. How does one achieve this?

Answer 1

Provided you leave the top-level namespace ninja empty, this is already natively supported in Python 3.3 and up, see PEP 420; all you have to do is create the ninja directory and leave out the __init__.py file in that directory:

eclipse_workspace
|
>-Ninja_core
| |
| >-ninja
|   |
|   >-core
|       |
|       >-__init__.py
|
>-Ninja_client
| |
| >-ninja
|   |
|   >-client
|       |
|       >-__init__.py

The ninja directories remain empty apart from the child core and client directories. Those directories do have __init__.py files.

In earlier Python versions, you can add support for namespaces by using setuptools namespace packages. The Zope and Plone projects have been releasing namespaced packages using setuptools for years now.

The basic principle is that you make sure your projects are packages with setup.py file, and are either installed in development mode or as end-products. Your ninja directories do then have a __init__.py file, but these must contain the following line only:

__import__('pkg_resources').declare_namespace(__name__)

and the setup.py file for each project must declare the namespace:

setup(
    # ...
    namespace_packages = ['ninja']
)