fast-ish python/jython IPC?

Question

All I want to do is make some RPC calls over sockets. I have a server that does backendish stuff running jython 2.5. I need to make some calls from a frontend server running Django on CPython. I've been beating my head against a wall getting any form of IPC going.

The list of things I've tried:

• Apache Thrift doesn't have any actual releases, just snapshots. I'd like to use something stable.
• JSON-RPC is interesting, and it should be able to run over sockets, but in practice most of the implementations only seem to work over HTTP. HTTP overhead is exactly what I'm trying to avoid.
• Protocol Buffers is really only a serialization protocol. From what I gather protobuf provides interface generation for RPC, but it's only the interface. Actually writing all the connection code is up to the user. If I'm going to be stuck using sockets, I'll just use JSON for serialization. It's simpler and faster.
• Pyro doesn't work properly with Jython as a server. Some sort of socket timeout issue. I've sent a message to the mailing list.
• pysage Yay for message passing! Only it requires python 2.6 or the processing module (which has compiled extensions). Jython is version 2.5 and doesn't allow compiled extensions.
• Candygram is an interesting alternative to pysage, but as far as I can tell it's unmaintained. I haven't even tried it out with Jython. Any experiences with it?
• Twisted Perspective Broker Twisted doesn't work on Jython.

I know that it'd be a snap doing this with XML-RPC, which makes me even more cranky. I want to avoid the overhead of HTTP, but at the same time I really don't want to get down and dirty with sockets to implement my own protocol. I'll do it wrong if I do.

Any ideas? I'm probably going to cry for about 20 minutes and then just use XML-RPC.

Answer 1

Have you considered Hessian? From the blurb:

The Hessian binary web service protocol makes web services usable without requiring a large framework, and without learning yet another alphabet soup of protocols. Because it is a binary protocol, it is well-suited to sending binary data without any need to extend the protocol with attachments.

It has Python client and Java server (and more besides).

Update: If you're dead against HTTP, why not just use SocketServer and pickle? Not much of a protocol needed, hard to get wrong. Send / receive pickled strings with length prefixes.

Answer 2

How about using sockets, but with the help of asyncore and asynchat?

Some links:

• An example
• http://docs.python.org/library/asyncore.html
• http://docs.python.org/library/asynchat.html

Answer 3

Two that look the most interesting to me:

• Gearman and Python bindings. It's quite a bit faster now that it's been re-written in C (originally perl). It's used in production (although I can't point to any examples of the python bindings being used in production). It has very interesting (to me) interfaces into MySQL and Postgresql. Finally, todays tweet from Django's Jacob Kaplan-Moss.

• RabbitMQ although because it's just a message queue you'll still have to serialize your own messages unless you also use celery.

Answer 4

My favorite.. zeroc's ice