Python 3.6.0 implicit namespace package

Question

I've found strange behavior with implicit namespace package in Python 3.6.0rc1. Could you please tell me if I am wrong or is it a Python 3.6 bug?

I am working with namespace package marrow which has two separated packages marrow.util and marrow.mailer. The second one depends on the first one.

Suppose we have marrow.util installed in site-packages for Python 2.7, 3.5 and 3.6:

$ ls -la /usr/lib/python*/site-packages/marrow
/usr/lib/python2.7/site-packages/marrow:
total 24
drwxr-xr-x.   3 root root  4096 Dec 23 12:23 .
drwxr-xr-x. 196 root root 16384 Dec 23 12:23 ..
drwxr-xr-x.   3 root root  4096 Dec 23 12:23 util

/usr/lib/python3.5/site-packages/marrow:
total 12
drwxr-xr-x.  3 root root 4096 Dec 23 12:24 .
drwxr-xr-x. 99 root root 4096 Dec 23 12:24 ..
drwxr-xr-x.  4 root root 4096 Dec 23 12:24 util

/usr/lib/python3.6/site-packages/marrow:
total 12
drwxr-xr-x.  3 root root 4096 Dec 23 14:25 .
drwxr-xr-x. 37 root root 4096 Dec 23 14:25 ..
drwxr-xr-x.  4 root root 4096 Dec 23 14:25 util

There are no __init__.py files here, which is correct because marrow is a namespace package. You can see this log message during installation:

Skipping installation of <deleted>/site-packages/marrow/__init__.py (namespace package)

And then you have the second part of marrow namespace package marrow.mailer built (but not installed) in some other directory. For example like this:

$ pwd
/builddir/build/BUILD/marrow.mailer-4.0.2

$ ls
coverage.xml debuglinks.list elfbins.list LICENSE.txt marrow.mailer.egg-info README.textile setup.py debugfiles.list debugsources.list example marrow PKG-INFO setup.cfg test

$ ls marrow/
__init__.py  __init__.pyc  mailer  __pycache__

When I run Python 2.7.12 or 3.5.2 in this folder and try to import marrow.util (from site-packages) it works as expected.

$ pwd
/builddir/build/BUILD/marrow.mailer-4.0.2

$ python2
Python 2.7.12 (default, Sep 29 2016, 12:52:02) 
[GCC 6.2.1 20160916 (Red Hat 6.2.1-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import marrow.util
>>>

$ python3.5
Python 3.5.2 (default, Sep 14 2016, 11:28:32) 
[GCC 6.2.1 20160901 (Red Hat 6.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import marrow.util
>>>

But when I try to import the same module with Python 3.6 it fails:

$ python3.6
Python 3.6.0rc1 (default, Dec 10 2016, 14:50:33) 
[GCC 6.2.1 20160916 (Red Hat 6.2.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import marrow.util
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'marrow.util'
>>> 

I've found this issue when I tried to build marrow.mailer as RPM package in Mock. With Python 2.7 and 3.5 everything works but Python 3.6 cannot import marrow.util from site-packages and for this reason tests of marrow.mailer fails during the build of RPM.

Example traceback from failed tests:

Traceback:
test/test_addresses.py:8: in <module>
    from marrow.mailer.address import Address, AddressList, AutoConverter
marrow/mailer/__init__.py:12: in <module>
    from marrow.mailer.message import Message
marrow/mailer/message.py:21: in <module>
    from marrow.mailer.address import Address, AddressList, AutoConverter
marrow/mailer/address.py:12: in <module>
    from marrow.util.compat import basestring, unicode, unicodestr, native
E   ModuleNotFoundError: No module named 'marrow.util'

I cannot find anything related to this issue in Changelog for Python 3.6.

Thanks for any help.

EDIT: I've checked sys.path in Python 3.6 and everything looks good:

$ python3.6
Python 3.6.0rc1 (default, Dec 10 2016, 14:50:33) 
[GCC 6.2.1 20160916 (Red Hat 6.2.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib64/python36.zip', '/usr/lib64/python3.6', '/usr/lib64/python3.6/lib-dynload', '/usr/lib64/python3.6/site-packages', '/usr/lib/python3.6/site-packages']
>>>

EDIT 2:

Because I still cannot find any solution and I haven't any response, I've created a simple Bash script which can reproduce my situation. The only thing you need is Python 3.5 and Python 3.6.

#!/bin/bash

# Change this to run script with different Python
#PYTHON=python3.5                   # system Python 3.5
PYTHON=~/temp/Python-3.6.0/python # compiled Python 3.6

# Create venv and activate
$PYTHON -m venv venv
source ./venv/bin/activate

# Install marrow.util package as a part of namespace package marrow
pip install marrow.util

# Create simple folder structure
mkdir -p marrow/mailer

# Create structure of __init__.py files
# For namespace package with related content
cat >> marrow/__init__.py << EOL
try: # pragma: no cover
    __import__('pkg_resources').declare_namespace(__name__)
except ImportError: # pragma: no cover
    __import__('pkgutil').extend_path(__path__, __name__)
EOL

# For mailer module just with print()
cat >> marrow/mailer/__init__.py << EOL
print('Imported!!!')
EOL

# Testing
# Importing marrow.util installed via pip in venv
$PYTHON -c "import marrow.util"
# Importing marrow.mailer created manually in PWD
$PYTHON -c "import marrow.mailer"

# deactivate venv
deactivate

If you execute this script with Python 3.5 you will see that Python 3.5 can import marrow.util installed via pip but it cannot import marrow.mailer in the local folder. But Python 3.6 can import local module marrow.mailer and it cannot import module marrow.util.

Answer 1

These packages are not using implicit namespacing ("native namespaces"), or if you do have versions that are, pin your dependencies to ensure you do not mix old and new style namespaces. They are utterly incompatible approaches.

You seem to be attempting, in your MCVE example code, to be constructing a namespace package (marrow/__init__.py declares via the old Python 2 explicit declaration substitution trick), A.K.A. pkg-resources-style namespace packages. This requires an argument to setup.py (actual packaging) and installation of metadata through package installation. Specifically, this method involves .pth file tricks (look in $VIRTUAL_ENV/lib/python3.?/site-packages) if installed "in development" and extraction/unpacking into that path for installation. Without that, there is not really a namespace, and your code won't be found, not with this older style. (The first, the installed one, will win.)

In a REPL, you can import the namespace, e.g. import marrow, then examine marrow.__path__ to see what was found/included as a diagnostic aid; my current WIP virtual environment on this machine has m.package, m.schema, and m.interface, which makes sense as I've been building releases of those recently. The more modern approach to native namespacing does allow for more free-form mixing, with the idea that a folder without an __init__.py just is a namespace, automatically merging across the PYTHONPATH, but that is not how namespaces used to work, alas. (All participants need that stub __init__.py with no other code present at all namespace levels.)

I am in the process of modernizing the entire Marrow ecosystem (I've begun with a few already, as noted above) to eliminate Python 2 legacy and begin embracing new Python 3 structures and approaches, including modern namespacing. Major version bump for everything and dependencies should be kept pinned less than these versions for any code needing the old namespaces still or on Python 2.

(I'm reacquiring a local Python 3.6 and 3.5 to investigate further.)