Alternatives to imp.find_module?

Question

Background

I've grown tired of the issue with pylint not being able to import files when you use namespace packages and divide your code-base into separate folders. As such I started digging into the astNG source-code which has been identified as the source of the trouble (see bugreport 8796 on astng). At the heart of the issue seems to be the use of pythons own imp.find_module in the process of finding imports.

What happens is that the import's first (sub)package - a in import a.b.c - is fed to find_module with a None path. Whatever path comes back is then fed into find_module the next pass in the look up loop where you try to find b in the previous example.

Pseudo-code from logilab.common.modutils:

path = None
while import_as_list:
      try:
           _, found_path, etc = find_module(import_as_list[0], path)
      #exception handling and checking for a better version in the .egg files
      path = [found_path]
      import_as_list.pop(0)

The Problem

This is what's broken: you only get the first best hit from find_module, which may or may not have your subpackages in it. If you DON'T find the subpackages, you have no way to back out and try the next one.

I tried explicitly using sys.path instead of None, so that the result could be removed from the path list and a second attempt be made, but python's module finder is clever enough that there doesn't have to be an exact match in the paths, making this approach unusable - to the best of my knowledge anyway.

Teary-eyed Plea

Is there an alternative to find_modules which will return ALL possible matches or take an exclude list? I'm also open to completely different solutions. Preferably not patching python by hand, but it wouldn't be impossible - at least for a local solution.

(Caveat emptor: I'm running python 2.6 and for reasons of current company policy can't upgrade, suggestions for p3k etc won't get marked as accepted unless it's the only answer.)

Answer 1

Since Python 2.5, the right way to do this is with pkgutil.iter_modules() (for a flat list) or pkgutil.walk_packages() (for a subpackage tree). Both are fully compatible with namespace packages.

For example, if I wanted to find just the subpackages/submodules of 'jmb', I would do:

import jmb, pkgutil
for (module_loader, name, ispkg) in pkgutil.iter_modules(jmb.__path__, 'jmb.'):
    # 'name' will be 'jmb.foo', 'jmb.bar', etc.
    # 'ispkg' will be true if 'jmb.foo' is a package, false if it's a module

You can also use iter_modules or walk_packages to walk all the modules on sys.path; see the docs linked above for details.

Answer 2

I've grown tired of this limitation in PyLint too.

I don't know a replacement for imp.find_modules(), but I think I found another way to deal with namespace packages in PyLint. See my comment on the bug report you linked to (http://www.logilab.org/ticket/8796).

The idea is to use pkg_resources to find namespace packages. Here's my addition to logilab.common.modutils._module_file(), just after while modpath:

  while modpath:
      if modpath[0] in pkg_resources._namespace_packages and len(modpath) > 1:
          module = sys.modules[modpath.pop(0)]
          path = module.__path__

This not very refined and only handles top-level namespace packages though.