Predefined Protocol for Communicating Over stdin/stdout/pipes in Python?

Question

I have a process that creates several subprocesses with which it needs to coummunicate in Python 2.5. I use the subprocess module to start the processes, setting stdin and stdout to subprocess.PIPE. So far so good.

In a loop, I then, run select.select() on each subproccess's stdout stream waiting for them to tell me they're ready. I then write to their stdin to give them work and repeat the process.

At the moment, I am simply using stdin.write and stdout.readline to preform the communication. However, I'd like to be able to communicate complex messages between the processes. Simple newline terminated messages won't suffice (unless I take strides to escape new lines in the messages somehow). I am thinking I can prefix all messages with the length in bytes so that my messages look something like:

6:foobar

But this brings me to my question: does something like this already exist? I don't really want to reinvent the wheel here. I want a library that tells me when there's a complete message ready and hands it over to me. There's plenty of other protocols that do things like this in various ways (TCP, Message Queue Servers, HTTP, etc) but they're all overkill for my use case. What's the right way to do message passing between processes in Python?

Answer 1

You could use json or pickle to encode the data structure that contains the message, and then use the same on the other side to read the message.

Answer 2

There are frameworks that really help you to address this kind of functionalities. If you want to send more complex messages without bother about sync issues. Some examples of really good frameworks are:

• 0mq or
• PYthon Remote Objects (Pyro)

These frameworks, mostly, work on top of TCP/IP. Slightly different approach to your stuff. But, maybe something you should consider.