How to organise the file structure of my already working plugin system?

Question

I am working on a project whose main design guiding principle is extensibility.

I implemented a plugin system by defining a metaclass that register - with a class method - the class name of any plugin that gets loaded (each type of plugin inherit from a specific class defined in the core code, as there are different types of plugins in the application). Basically this means that a developer will have to define his class as

class PieChart(ChartPluginAncestor):
    # Duck typing:
    # Implement compulsory methods for Plugins 
    # extending Chart functionality

and the main program will know of his presence because PieChart will be included in the list of registered plugins available at ChartPluginAncestor.plugins.

Being the mounting method a class method, all plugins get registered when their class code is loaded into memory (so even before an object of that class is instantiated).

The system works good enough™ for me (although I am always open to suggestions on how to improve the architecture!) but I am now wondering what would be the best way to manage the plugin files (i.e. where and how the files containing the plugins should be stored).

So far I am using - for developing purposes - a package that I called "plugins". I put all my *.py files containing plugins classes in the package directory, and I simply issue import plugins in the main.py file, for all the plugins to get mounted properly.

EDIT: Jeff pointed out in the comments that import plugins the classes contained in the various modules of the packages won't be readily available (I did not realise this as I was - for debugging purposes - importing each class separately with from plugins.myAI import AI).

However this system is only good while I am developing and testing the code, as:

• Plugins might come with their own unittests, and I do not want to load those in memory.
• All plugins are currently loaded into memory, but indeed there are certain plugins which are alternative versions of the same feature, so you really just need to know that you can switch between the two, but you want to load into memory just the one you picked from the config pane.
• At some point, I will want to have a double location for installing plugins: a system-wide location (for example somewhere under /usr/local/bin/) and a user-specific one (for example somewhere under /home/<user>/.myprogram/).

So my questions are really - perhaps - three:

1. Plugin container: what is the most sensible choice for my goal? single files? packages? a simple directory of .py files?)
2. Recognise the presence of plugins without necessarily loading (importing) them: what is a smart way to use Python introspection to do so?
3. Placing plugins in two different locations: is there a standard way / best practice (under gnu/linux, at least) to do that?

Answer 1

The question is hard to address, because the needs are complex. Anyway I will try with some suggestions.

About

Placing plugins in two different locations: is there a standard way / best practice (under gnu/linux, at least) to do that?

A good approach is virtualenv. Virtualenv is a python module to build "isolated" python installation. It is the better way to get separate projects working together. You get a brand new site-package where you can put your plugins with the relevant project modules.

Give it a try: http://pypi.python.org/pypi/virtualenv

Plugin container: what is the most sensible choice for my goal? single files? packages? a simple directory of .py files?)

A good approach is a python package which can do a "self registration" upon import: simply define inside the package directory a proper init.py

An example can be http://www.qgis.org/wiki/Writing_Python_Plugins and also the API described here http://twistedmatrix.com/documents/current/core/howto/plugin.html

See also http://pypi.python.org/pypi/giblets/0.2.1

Giblets is a simple plugin system based on the component architecture of Trac. In a nutshell, giblets allows you to declare interfaces and discover components that implement them without coupling.

Giblets also includes plugin discovery based on file paths or entry points along with flexible means to manage which components are enabled or disabled in your application.

Answer 2

I also have a plugin system with three types of plugins, though I don't claim to have done it well. You can see some details here.

For internal plugins, I have a package (e.g., MethodPlugins) and in this package is a module for each plugin (e.g., MethodPlugins.IRV). Here is how I load the plugins:

1. Load the package (import MethodPlugins)

2. Use pkgutil.iter_modules to load all the modules there (e.g., MethodPlugins.IRV)

3. All the plugins descend from a common base class so I can use __subclassess__ to identify them all.

I believe this would allow you to recognize plugins without actually loading them, though I don't do that as I just load them all.

For external plugins, I have a specified directory where users can put them, and I use os.listdir to import them. The user is required to use the right base class so I can find them.

I would be interested in improving this as well, but it also works good enough for me. :)