What is the correct setup for namespace packages, both installed and as source package? [duplicate]

Question

In Python, a namespace package allows you to spread Python code among several projects. This is useful when you want to release related libraries as separate downloads. For example, with the directories Package-1 and Package-2 in PYTHONPATH,

Package-1/namespace/__init__.py
Package-1/namespace/module1/__init__.py
Package-2/namespace/__init__.py
Package-2/namespace/module2/__init__.py

the end-user can import namespace.module1 and import namespace.module2.

What's the best way to define a namespace package so more than one Python product can define modules in that namespace?

Answer 1

TL;DR:

On Python 3.3 you don't have to do anything, just don't put any __init__.py in your namespace package directories and it will just work. On pre-3.3, choose the pkgutil.extend_path() solution over the pkg_resources.declare_namespace() one, because it's future-proof and already compatible with implicit namespace packages.

---

Python 3.3 introduces implicit namespace packages, see PEP 420.

This means there are now three types of object that can be created by an import foo:

• A module represented by a foo.py file
• A regular package, represented by a directory foo containing an __init__.py file
• A namespace package, represented by one or more directories foo without any __init__.py files

Packages are modules too, but here I mean "non-package module" when I say "module".

First it scans sys.path for a module or regular package. If it succeeds, it stops searching and creates and initalizes the module or package. If it found no module or regular package, but it found at least one directory, it creates and initializes a namespace package.

Modules and regular packages have __file__ set to the .py file they were created from. Regular and namespace packages have __path__set to the directory or directories they were created from.

When you do import foo.bar, the above search happens first for foo, then if a package was found, the search for bar is done with foo.__path__as the search path instead of sys.path. If foo.bar is found, foo and foo.bar are created and initialized.

So how do regular packages and namespace packages mix? Normally they don't, but the old pkgutil explicit namespace package method has been extended to include implicit namespace packages.

If you have an existing regular package that has an __init__.py like this:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

... the legacy behavior is to add any other regular packages on the searched path to its __path__. But in Python 3.3, it also adds namespace packages.

So you can have the following directory structure:

├── path1
│   └── package
│       ├── __init__.py
│       └── foo.py
├── path2
│   └── package
│       └── bar.py
└── path3
    └── package
        ├── __init__.py
        └── baz.py

... and as long as the two __init__.py have the extend_path lines (and path1, path2 and path3 are in your sys.path) import package.foo, import package.bar and import package.baz will all work.

pkg_resources.declare_namespace(__name__) has not been updated to include implicit namespace packages.

Answer 2

There's a standard module, called pkgutil, with which you can 'append' modules to a given namespace.

With the directory structure you've provided:

Package-1/namespace/__init__.py
Package-1/namespace/module1/__init__.py
Package-2/namespace/__init__.py
Package-2/namespace/module2/__init__.py

You should put those two lines in both Package-1/namespace/__init__.py and Package-2/namespace/__init__.py (*):

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

(* since -unless you state a dependency between them- you don't know which of them will be recognized first - see PEP 420 for more information)

As the documentation says:

This will add to the package's __path__ all subdirectories of directories on sys.path named after the package.

From now on, you should be able to distribute those two packages independently.

Answer 3

This is an old question, but someone recently commented on my blog that my posting about namespace packages was still relevant, so thought I would link to it here as it provides a practical example of how to make it go:

https://web.archive.org/web/20150425043954/http://cdent.tumblr.com/post/216241761/python-namespace-packages-for-tiddlyweb

That links to this article for the main guts of what's going on:

http://www.siafoo.net/article/77#multiple-distributions-one-virtual-package

The __import__("pkg_resources").declare_namespace(__name__) trick is pretty much drives the management of plugins in TiddlyWeb and thus far seems to be working out.

Answer 4

This section should be pretty self-explanatory.

In short, put the namespace code in __init__.py, update setup.py to declare a namespace, and you are free to go.