What are prevalent techniques for enabling user code extensions in Python?

Question

I'm looking for techniques that allow users to override modules in an application or extend an application with new modules.

Imagine an application called pydraw. It currently provides a Circle class, which inherits Shape. The package tree might look like:

/usr/lib/python/
└── pydraw
    ├── __init__.py
    ├── shape.py
    └── shapes
        ├── circle.py
        └── __init__.py

Now suppose I'd like to enable dynamic discovery and loading of user modules that implement a new shape, or perhaps even the Shape class itself. It seems most straightforward for a user's tree to have the same structure as the application tree, such as:

/home/someuser/python/
└── pydraw
    ├── __init__.py
    ├── shape.py       <-- new superclass
    └── shapes
        ├── __init__.py
        └── square.py   <-- new user class

In other words, I'd like to overlay and mask an application tree with same-named files from the user's tree, or at least get that apparent structure from a Python point of view.

Then, by configuring sys.path or PYTHONPATH, pydraw.shapes.square might be discoverable. However, Python's module path search doesn't find modules such as square.py. I presume this is because __method__ already contains a parent module at another path.

How would you accomplish this task with Python?

Answer 1

Discovery of extensions can be a bit brittle and complex, and also requires you to look through all of PYTHONPATH which can be very big.

Instead, have a configuration file that lists the plugins that should be loaded. This can be done by listing them as module names, and also requiring that they are located on the PYTHONPATH, or by simply listing the full paths.

If you want per user level configurations, I'd have both a global configuration file that lists modules, and a per user one, and just read these config files instead of trying some discovery mechanism.

Also you are trying to not only add plugins, but override components of your application. For that I would use the Zope Component Architecture. It is however not yet fully ported to Python 3, but it's designed to be used in exactly this kinds of cases, although from your description this seems to be a simple case, and the ZCA might be overkill. But look at it anyway.

• http://www.muthukadan.net/docs/zca.html
• http://pypi.python.org/pypi/zope.component

Answer 2

If you want to load python code dynamically from different locations, you can extend the search __path__ attributes by using the pkgutil module:

By placing these lines into each pydraw/__init__.py and pydraw/shapes/__init__.py:

from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

You will be able to write import statement as if you had a unique package:

>>> import pydraw.shapes
>>> pydraw.shapes.__path__
['/usr/lib/python/pydraw/shapes', '/home/someuser/python/pydraw/shapes']
>>> from pydraw.shapes import circle, square
>>>

You may think about auto-registration of your plugins. You can still use basic python code for that by setting a module variable (which will act as a kind of singleton pattern).

Add the last line in every pydraw/shapes/__init__.py file:

 from pkgutil import extend_path
 __path__ = extend_path(__path__, __name__)

 # your shape registry
 __shapes__ = []

You can now register a shape in top of its related module (circle.py or square.py here).

 from pydraw.shapes import __shapes__
 __shapes__.append(__name__)

Last check:

 >>> from pydraw.shapes import circle,square
 >>> from pydraw.shapes import circle,square,__shapes__
 >>> __shapes__
 ['pydraw.shapes.circle', 'pydraw.shapes.square']

Answer 3

A method I used to handle such a problem was with a provider pattern.

In your module shape.py :

class BaseShape:
   def __init__(self):
      pass
provider("BaseShape", BaseShape)

and in the user's shape.py module :

class UserBaseShape:
   def __init__(self):
      pass
provider("BaseShape", UserBaseShape)

with provider method doing something like this :

def provider(provide_key, provider_class):
   global_providers[provide_key] = provider_class

And when you need to instanciate an object, use a ProvideFactory like this

class ProvideFactory:
   def get(self, provide_key, *args, **kwargs):
      return global_providers[provide_key](*args, **kwargs)