Pluggable Python sub-command pattern?

Question

I'm looking for a good pattern for how to implement Python sub-commands, where the main command looks up the subcommand at run time (instead of knowing the list of all possible sub-commands; this allows the "application" to be easily extended with new sub-commands without having to change the main code)

E.g:

 topcmd.py foo

will look in /some/dir for foo.py and if it exists, run it. Or some variation of it.

The code invoked in foo.py should preferably be a well-defined function or method on a class or object.

Answer 1

While this question is actually quite broad, there are sufficient tools available within a typical default Python installation (i.e. with setuptools) that this is relatively achievable, in a way that is actually extensible so that other packages can be created/installed in a manner that provide new, discoverable subcommands for your main program.

Your base package can provide a standard entry_point in the form of console_scripts that point to your entry point that will feed all arguments into an instance of some argument parser (such as argparse), and some kind of registry which you might implement under a similar scheme as console_scripts, except under your specific entry_points group so that it would iterate through every entry and instantiate the objects that would also provide their own ArgumentParser instances which your main entry point would dynamically register to itself as a subcommand, thus showing your users what subcommands are actually available and what their invocation might be like.

To provide an example, in your main package's setup.py, you might have an entry like

setup(
    name='my.package',
    # ...
    entry_points={
        'console_scripts': [
            'topcmd = my.package.runtime:main',
        ],
        'my.package.subcmd': [
            'subcmd1 = my.package.commands:subprog1',
            'subcmd2 = my.package.commands:subprog2',
        ],
    },
    # ...
)

Inside the my/package/runtime.py source file, a main method will have to construct a new ArgumentParser instance, and while iterating through the entry points provided by pkg_resources.working_set, for example:

from pkg_resources import working_set

def init_parser(argparser):  # pass in the argparser provided by main
    commands = argparser.add_subparsers(dest='command')
    for entry_point in working_set.iter_entry_points('my.package.subcmd'):
        subparser = commands.add_parser(entry_point.name)
        # load can raise exception due to missing imports or error in object creation
        subcommand = entry_point.load()
        subcommand.init_parser(subparser) 

So in the main function, the argparser instance it created could be passed into a function like one in above, and the entry point 'subcmd1 = my.package.commands:subprog1' will be loaded. Inside my/package/command.py, an implemented init_parser method must be provided, which will take the provided subparser and populate it with the required arguments:

class SubProgram1(object):
    def init_parser(self, argparser)
        argparser.add_argument(...) 

subprog1 = SubProgram1() 

Oh, one final thing, after passing in the arguments to the main argparser.parse_args(...), the name of the command is provided to argparser.command. It should be possible to change that to the actual instance, but that may or may not achieve what you exactly want (because the main program might want to do further work/validation before actually using the command). That part is another complicated part, but at least the argument parser should contain the information required to actually run the correct subprogram.

Naturally, this includes absolutely no error checking, and it must be implemented in some form to prevent faulty subcommand classes from blowing up the main program. I have made use of a pattern like this one (albeit with a lot more complex implementation) that can support an arbitrary amount of nested subcommand. Also packages that want to implement custom commands can simply add their own entry to the entry point group (in this case, to my.package.subcmd) for their own setup.py. For example:

setup(
    name="some.other.package",
    # ...
    entry_points={
        'my.package.subcmd': [
            'extracmd = some.other.package.commands:extracmd',
        ],
    },
    # ...
)

Addendum:

As requested, an actual implementation that's used in production is in a package (calmjs) that I currently maintain. Installing that package (into a virtualenv) and running calmjs on the command line should show a listing of subcommands identical to the entries defined in the main package's entry points. Installing an additional package that extends the functionality (such as calmjs.webpack) and running calmjs again will now list calmjs.webpack as an additional subcommand.

The entry points references instances of subclasses to the Runtime class, and in it there is a place where the subparser is added and if satisfies registration requirements (many statements following that relate to various error/sanity checking, such as what to do when multiple packages define the same subcommand name for runtime instances, amongst other things), registered to the argparser instance on that particular runtime instance, and the subparser is passed into the init_argparser method of the runtime that encapsulates the subcommand. As an example, the calmjs webpack subcommand subparser is set up by its init_argparser method, and that package registers the webpack subcommand in its own setup.py. (To play with them, please just simply use pip to install the relevant packages).

Answer 2

You can use the __import__ function to dynamically import a module using a string name passed on the command line.

mod = sys.argv[1]
command =__import__(mod)
# assuming your pattern has a run method defined. 
command.run()

Error handling, etc left as an exercise for the reader

Edit: This would depend on user plugins being installed via pip. If you want users to drop plugins into a folder without installing, then you would have to add that folder to your python path.