What is the difference between `sys.meta_path` and `sys.path_hooks` importer objects?

Question

Using importlib, what is the difference between "Meta Path Finder" (found by traversing over sys.meta_path) and Path Entry Finder" (found by traversing over sys.path_hooks)?

The first type is called upon begin of an import, but when is the second type used? Do both return a spec object?

I want to implement a customized import, where a module can be imported from sources other than *.py or *.pyc, e.g. from a stream. How can this be done?

Answer 1

When a module is to be imported, the interpreter first walks through the list of objects in sys.meta_path, calling the find_spec() or (deprecated since 3.4) find_module() method on each. )The interface is documented in the importlib.abc.MetaPathFinder abstract base class.) These are queried before any other importers (including frozen and built-in) are checked and so can override any other import processing.

The PathFinder object in sys.meta_path is what uses sys.path and sys.path_hooks. (Except in Python < 3.4 where the PathFinder functionality is built in to the interpreter to be used when nothing in sys.meta_path can load a module.)

The PathFinder walks through the list of paths in sys.path. For each path, if finder is not already cached for that path in sys.path_importer_cache it walks through the list of callables in sys.path_hooks, calling each one with the path to see if it will produce a finder; it caches the first one it finds in sys.path.importer_cache_.

Once it's got the finder it queries that via the find_spec() or deprecated find_module() method to see if it can find that module. If so, it can continue on to import it, otherwise it starts the step above with the next path on sys.path.

This was initially described in PEP 302, but PEP 451 is pretty much the modern behaviour; the importlib documentation appears to be the current spec.

There's considerably more detail summarized (with more links) in my personal notes.

Answer 2

sys.path_hooks returns a finder factory.

Path hooks are called as part of sys.path (or package.__path__ ) processing

as we read in PEP 302 relevant part which you should read to do what you want.

Coming to speak of, we use a custom hook in my code but I would not recommend you to copy it verbatim (I am really not sure about the hocus pocus we do with init files)

However the process is a bit like in there - the find_module method returns self or None depending on what you want to accept as a module and the load_module method proceeds to load that by compiling the code and assigning it an entry into sys.modules. By replacing those parts you can pretty much load whatever you want.

Related:

• Package-specific import hooks in Python
• Python sys.path_hooks Examples