Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How do you recursively get all submodules in a python package?

Problem

I have a folder structure like this:

- modules
    - root
        - abc
            hello.py
            __init__.py
        - xyz
            hi.py
            __init__.py
          blah.py
          __init__.py
      foo.py
      bar.py
      __init_.py

Here is the same thing in string format:

"modules",
"modues/__init__.py",
"modules/foo.py",
"modules/bar.py",
"modules/root",
"modules/root/__init__.py",
"modules/root/blah,py",
"modules/root/abc",
"modules/root/abc/__init__.py",
"modules/root/abc/hello.py",
"modules/root/xyz",
"modules/root/xyz/__init__.py",
"modules/root/xyz/hi.py"

I am trying to print out all the modules in the python import style format. An example output would like this:

modules.foo
modules.bar
modules.root.blah
modules.root.abc.hello
modules.root.xyz.hi

How can I do this is in python(if possible without third party libraries) easily?

What I tried

Sample Code

import pkgutil

import modules

absolute_modules = []


def find_modules(module_path):
    for package in pkgutil.walk_packages(module_path):
        print(package)
        if package.ispkg:
            find_modules([package.name])
        else:
            absolute_modules.append(package.name)


if __name__ == "__main__":
    find_modules(modules.__path__)
    for module in absolute_modules:
        print(module)

However, this code will only print out 'foo' and 'bar'. But not 'root' and it's sub packages. I'm also having trouble figuring out how to convert this to preserve it's absolute import style. The current code only gets the package/module name and not the actual absolute import.

like image 957
daegontaven Avatar asked Feb 20 '18 06:02

daegontaven


People also ask

How do I see what Python modules are in a package?

Inside the package, you can find your modules by directly using __loader__ of course. Show activity on this post. That only works for modules, not packages. Try it on Python's logging package to see what I mean.

What is __ all __ in Python?

The __all__ in Python is a list of strings defining what symbols in a module will be exported when from <module> import * is used on the module.

How do you import all functions in Python?

We can use import * if we want to import everything from a file in our code. We have a file named functions.py that contains two functions square() and cube() . We can write from functions import * to import both functions in our code. We can then use both square() and cube() functions in our code.

Which operator is used in the Python to import all modules from packages?

We can import modules from packages using the dot (.) operator.


2 Answers

This uses setuptools.find_packages (for the packages) and pkgutil.iter_modules for their submodules. Python2 is supported as well. No need for recursion, it's all handled by these two functions used together.

import sys
from setuptools import find_packages
from pkgutil import iter_modules

def find_modules(path):
    modules = set()
    for pkg in find_packages(path):
        modules.add(pkg)
        pkgpath = path + '/' + pkg.replace('.', '/')
        if sys.version_info.major == 2 or (sys.version_info.major == 3 and sys.version_info.minor < 6):
            for _, name, ispkg in iter_modules([pkgpath]):
                if not ispkg:
                    modules.add(pkg + '.' + name)
        else:
            for info in iter_modules([pkgpath]):
                if not info.ispkg:
                    modules.add(pkg + '.' + info.name)
    return modules
like image 150
rwst Avatar answered Oct 03 '22 09:10

rwst


So I finally figured out how to do this cleanly and get pkgutil to take care of all the edge case for you. This code was based off python's help() function which only displays top level modules and packages.

import importlib
import pkgutil

import sys

import modules


def find_abs_modules(module):
    path_list = []
    spec_list = []
    for importer, modname, ispkg in pkgutil.walk_packages(module.__path__):
        import_path = f"{module.__name__}.{modname}"
        if ispkg:
            spec = pkgutil._get_spec(importer, modname)
            importlib._bootstrap._load(spec)
            spec_list.append(spec)
        else:
            path_list.append(import_path)
    for spec in spec_list:
        del sys.modules[spec.name]
    return path_list


if __name__ == "__main__":
    print(sys.modules)
    print(find_abs_modules(modules))
    print(sys.modules)

This will work even for builtin packages.

like image 24
daegontaven Avatar answered Oct 03 '22 09:10

daegontaven