Lazy-loading modules in python

Question

I'm trying to put together a system that will handle lazy-loading of modules that don't explicitly exist. Basically I have an http server with a number of endpoints that I don't know ahead of time that I would like to programmatically offer for import. These modules would all have a uniform method signature, they just wouldn't exist ahead of time.

import lazy.route as test
import lazy.fake as test2

test('Does this exist?')  # This sends a post request.
test2("This doesn't exist.")  # Also sends a post request

I can handle all the logic I need around these imports with a uniform decorator, I just can't find any way of "decorating" imports in python, or actually interacting with them in any kind of programmatic way.

Does anyone have experience with this? I've been hunting around, and the closest thing I've found is the ast module, which would lead to a really awful kind of hacky implementation in my current under my current understanding (something like finding all import statements and manually over-writing the import function)

Not looking for a handout, just a piece of the python codebase to start looking at, or an example of someone that's done something similar.

Answer 1

I got a little clever in my googling and managed to find a PEP that specifically addressed this issue, it just happens to be relatively unknown, probably because the subset of reasonable uses for this is pretty narrow.

I found an excellent piece of example code showing off the new sys.meta_path implementation. I've posted it below for information on how to dynamically bootstrap your import statements.

import sys


class VirtualModule(object):

   def hello(self):
      return 'Hello World!'


class CustomImporter(object):

   virtual_name = 'my_virtual_module'

   def find_module(self, fullname, path):
      """This method is called by Python if this class
         is on sys.path. fullname is the fully-qualified
         name of the module to look for, and path is either
         __path__ (for submodules and subpackages) or None (for
         a top-level module/package).

         Note that this method will be called every time an import
         statement is detected (or __import__ is called), before
         Python's built-in package/module-finding code kicks in."""

      if fullname ==  self.virtual_name:

         # As per PEP #302 (which implemented the sys.meta_path protocol),
         # if fullname is the name of a module/package that we want to
         # report as found, then we need to return a loader object.
         # In this simple example, that will just be self.

         return self

      # If we don't provide the requested module, return None, as per
      # PEP #302.

      return None

   def load_module(self, fullname):
      """This method is called by Python if CustomImporter.find_module
         does not return None. fullname is the fully-qualified name
         of the module/package that was requested."""

      if fullname != self.virtual_name:
         # Raise ImportError as per PEP #302 if the requested module/package
         # couldn't be loaded. This should never be reached in this
         # simple example, but it's included here for completeness. :)
         raise ImportError(fullname)

      # PEP#302 says to return the module if the loader object (i.e,
      # this class) successfully loaded the module.
      # Note that a regular class works just fine as a module.
      return VirtualModule()


if __name__ == '__main__':

   # Add our import hook to sys.meta_path
   sys.meta_path.append(CustomImporter())

   # Let's use our import hook
   import my_virtual_module
   print my_virtual_module.hello()

The full blog post is here