The "right" way to add python scripting to a non-python application

Question

I'm currently in the process of adding the ability for users to extend the functionality of my desktop application (C++) using plugins scripted in python.

The naive method is easy enough. Embed the python static library and follow any number of the dozens of tutorials scattered around the web describing how to initialize and call python files, and you're pretty much done.

However...

What I'm looking for is more like what Blender does. Blender is completely customizable through python scripts, and it requires an external python executable. (Ie. python isn't actually embedded in the blender executable at all.) So, naturally, you can include any modules you already have in your site-packages directory when you are writing blender scripts. Not that that's advised, since that would limit the portability of your script.

So, what I want to know is if there is already a way to have your cake and eat it too. I want a plugin system that uses:

• An embedded python interpreter.

  The downside of Blender's approach is that it forces you to have a specific, possibly outdated version of python installed globally on your system. Having an embedded interpreter allows me to control what version of python is being used.

• Firewall plugins.

  Some equivalent of a virtualenv for each plugin; allowing them to install all the modules they need or want, but keeping them seperated from possible conflicts in other plugins. Maybe zc.buildout is a better candidate here, but, again, I'm very open to suggestion. I'm a bit at a loss as to the best way to accomplish this.

• As painless as possible...

  For the user. I'm willing to go the extra mile, just so long as most of the above is as transparent to the plugin writer as possible.

---

If any of you folks out there have any experience with this sort of thing, your help would be much appreciated. :)

---

Edit: Basically, the short version of what I want is the simplicity of virtualenv, but without the bundled python interpreter, and a way to activate a specific "virtual environment" programmatically, like zc.buildout does with sys.path manipulation (the sys.path[0:0] = [...] trick).

Both virtualenv and zc.buildout contain portions of what I want, but neither produce relocatable builds that I, or a plugin developer can simply zip up and send to another computer.

Simply manipulating .pth files, or manipulating sys.path directly in a script, executed from my application gets me half-way there. But it is not enough when compiled modules are necessary, such as the PIL.

Answer 1

One effective way to accomplish this is to use a message-passing/communicating processes architecture, allowing you to accomplish your goal with Python, but not limiting yourself to Python.

------------------------------------ | App  <--> Ext. API <--> Protocol | <--> (Socket) <--> API.py <--> Script ------------------------------------ 

This diagram attempts to show the following: Your application communicates with external processes (for example Python) using message passing. This is efficient on a local machine, and can be portable because you define your own protocol. The only thing that you have to give your users is a Python library that implements your custom API, and communicates using a Send-Receive communication loop between your user's script and your application.

Define Your Application's External API

Your application's external API describes all of the functions that an external process must be able to interact with. For example, if you wish for your Python script to be able to draw a red circle in your application, your external API may include Draw(Object, Color, Position).

Define A Communication Protocol

This is the protocol that external processes use to communicate with your application through it's external API. Popular choices for this might be XML-RPC, SunRPC, JSON, or your own custom protocol and data format. The choice here needs to be sufficient for your API. For example, if you are going to be transferring binary data then JSON might require base64 encoding, while SunRPC assumes binary communication.

Build Your Application's Messaging System

This is as simple as an infinite loop receiving messages in your communication protocol, servicing the request within your application, and replying over the same socket/channel. For example, if you chose JSON then you would receive a message containing instructions to execute Draw(Object, Color, Position). After executing the request, you would reply to the request.

Build A Messaging Library For Python (or whatever else)

This is even simpler. Again, this is a loop sending and receiving messages on behalf the library user (i.e. your users writing Python scripts). The only thing this library must do is provide a programmatic interface to your Application's External API and translate requests into your communication protocol, all hidden from your users.

Using Unix Sockets, for example, will be extremely fast.

Plugin/Application Rendezvous

A common practice for discovering application plugins is to specify a "well known" directory where plugins should be placed. This might be, for example:

~/.myapp/plugins 

The next step is for your application to look into this directory for plugins that exist. Your application should have a some smarts to be able to distinguish between Python scripts that are, and are not, real scripts for your application.

Let's assume that your communication protocol specifies that each script will communicate using JSON over StdInput/StdOuput. A simple, effective approach is to specify in your protocol that the first time a script runs it sends a MAGIC_ID to standard out. That is, your application reads the first, say, 8 bytes, and looks for a specific 64-bit value that identifies it as a script.

Additionally, you should include in your External API methods that allow your scripts to identify themselves. For example, a script should be able to inform the application through the External API things such as Name, Description, Capabilities, Expectations, essentially informing the application what it is, and what it will be doing.

Answer 2

I don't see the problem with embedding Python with, for example, Boost.Python. You'll get everything you ask for:

• It will be embedded, and it will be an interpreter (with enough access to implement auto-completion and such)
• You can create a new interpreter per-script and have completely separated python environments
• ... and as transparent as it's possible

I mean, you'll still have to expose and implement an API, but 1) this is a good thing, 2) Blender does it too and 3) I really can't think of another way that leverages you from this work...

PS: I have little experience with Python / Boost.Python but have worked extensively with Lua / LuaBind, which is kinda the same