Building python dependency graph on the fly

Question

I want to determine the imports of each python module in a library for use in a custom configuration management framework. I've seen tools like snakefood, but I'd prefer to compute a dependency graph during normal program execution, instead of as a ~compilation step.

So far I've tried writing a custom Finder and Loader. Both approaches work as expected the first time a module is imported, but don't trigger on subsequent imports thanks to the sys.modules cache.

I can override __built__.__import__ for notifications every time a module is imported, but it seems like this approach is ill-advised since PEP 302.

Is there an import hook I can place in front of sys.modules cache lookups? Or another way to quickly compute dependencies on the fly?

Answer 1

It's possible (if hacky) to reassign to sys.modules:

import sys
import inspect

old_sys_modules = sys.modules

class NewSysModules():
  def __getitem__(self, mod_name):
    frame = inspect.currentframe().f_back
    while frame.f_globals["__name__"].startswith("importlib"):
      frame = frame.f_back # go back until we're not in a importlib frame
    importer = frame.f_globals["__name__"]
    print(f"importing {mod_name} from {importer}")

    return old_sys_modules[mod_name]

  def __setitem__(self, mod_name, module):
    old_sys_modules[mod_name] = module

sys.modules = NewSysModules()

However, this might require some maintenance if the import system changes.