__init__.py required for pkgutil.walk_packages in python3?

Question

PEP420 makes __init__.py files optional: https://docs.python.org/3/whatsnew/3.3.html#pep-420-implicit-namespace-packages

Though it seems without them, pkgutil.walk_packages does not function as desired: https://docs.python.org/3/library/pkgutil.html#pkgutil.walk_packages

Consider the following example:

$ tree foo
foo
├── bar
│   ├── baz.py
│   └── __init__.py
├── __init__.py
└── womp.py

And a test script

# test.py
import pkgutil

import foo


for _, mod, _ in pkgutil.walk_packages(foo.__path__, foo.__name__ + '.'):
    print(mod)

In both python2 and python3 I get the following output:

$ python2.7 test.py
foo.bar
foo.bar.baz
foo.womp
$ python3.5 test.py
foo.bar
foo.bar.baz
foo.womp

Removing the __init__.py files and only using python3, I get this:

$ find -name '__init__.*' -delete
$ python3.5 test.py
foo.bar

The modules are definitely importable:

$ python3.5 -c 'import foo.bar.baz'
$

Is this a bug? Am I forced to create the __init__.py files to achieve what I want?

Answer 1

As a workaround (maybe this will help someone else), I'm using something like this. It isn't perfect (broken if pwd changes or if the packages are not rooted at .) but it does do what I want to do for my simple usecase:

def walk_modules(pkg):
    assert hasattr(pkg, '__path__'), 'This function is for packages'
    path = pkg.__name__.replace('.', '/')
    modules = []
    for root, _, filenames in os.walk(path):
        for filename in filenames:
            if filename.startswith('.') or not filename.endswith('.py'):
                continue
            path = os.path.join(root, filename)
            modules.append(os.path.splitext(path)[0].replace('/', '.'))
    for module in sorted(modules):
        yield __import__(module, fromlist=['__trash'])

Answer 2

Another approach that respects the __path__ attribute for merged namespace packages:

import pkgutil
from pathlib import Path

def iter_packages(path, prefix, onerror=None):
    """ Find packages recursively, including PEP420 packages """
    yield from pkgutil.walk_packages(path, prefix, onerror)
    namespace_packages = {}
    for path_root in path:
        for sub_path in Path(path_root).iterdir():
            # TODO: filter to legal package names
            if sub_path.is_dir() and not (sub_path / '__init__.py').exists():
                ns_paths = namespace_packages.setdefault(prefix + sub_path.name, [])
                ns_paths.append(str(sub_path))
    for name, paths in namespace_packages.items():
        # TODO: construct a loader somehow?
        yield pkgutil.ModuleInfo(None, name, True)
        yield from iter_packages(paths, name + '.', onerror)