Loading each .py file in a path - imp.load_module complains about relative import

Question

I am trying to parse a given path for python source files, import each file and DoStuff™ to each imported module.

def ParsePath(path):
    for root, dirs, files in os.walk(path):
        for source in (s for s in files if s.endswith(".py")):
            name = os.path.splitext(os.path.basename(source))[0]
            m = imp.load_module(name, *imp.find_module(name, [root]))
            DoStuff(m)

The above code works, but packages aren't recognized ValueError: Attempted relative import in non-package

My question is basically, how do I tell imp.load_module that a given module is part of a package?

Answer 1

You cannot directly tell Importer Protocol method load_module that the module given is part of the package. Taken from PEP 302 New Import Hooks

The built-in __import__ function (known as PyImport_ImportModuleEx in import.c) will then check to see whether the module doing the import is a package or a submodule of a package. If it is indeed a (submodule of a) package, it first tries to do the import relative to the package (the parent package for a submodule). For example if a package named "spam" does "import eggs", it will first look for a module named "spam.eggs". If that fails, the import continues as an absolute import: it will look for a module named "eggs". Dotted name imports work pretty much the same: if package "spam" does "import eggs.bacon" (and "spam.eggs" exists and is itself a package), "spam.eggs.bacon" is tried. If that fails "eggs.bacon" is tried. (There are more subtleties that are not described here, but these are not relevant for implementers of the Importer Protocol.)

Deeper down in the mechanism, a dotted name import is split up by its components. For "import spam.ham", first an "import spam" is done, and only when that succeeds is "ham" imported as a submodule of "spam".

The Importer Protocol operates at this level of individual imports. By the time an importer gets a request for "spam.ham", module "spam" has already been imported.

You must then simulate what the built-in import does and load parent packages before loading sub modules.

Answer 2

The function imp.find_module always takes a plain module name without dots, but the documentation of imp.load_module says

The name argument indicates the full module name (including the package name, if this is a submodule of a package).

So you could try this:

def ParsePath(path):
    for root, dirs, files in os.walk(path):
        for source in (s for s in files if s.endswith(".py")):
            name = os.path.splitext(os.path.basename(source))[0]
            full_name = os.path.splitext(source)[0].replace(os.path.sep, '.')
            m = imp.load_module(full_name, *imp.find_module(name, [root]))
            DoStuff(m)