Efficient Python IPC [closed]

Question

I'm making an application in Python3, which will be divided in batch and gui parts. Batch is responsible for processing logic and gui is responsible for displaying it.

Which inter-process communication (IPC) framework should I use with the following requirements:

1. The GUI can be run on other device than batch (GUI can be run on the same device, on smartphone, tablet etc, locally or over network).
2. The batch (Python3 IPc library) should work with no problem on Linux, Mac, Windows, ...
3. The IPC should support GUI written in different languages (Python, Javascript, ...)
4. The performance of IPC is important - it should be as "interactive" as possible, but without losing information.
5. Several GUI could be connected to the same batch.

additional: Will the choice be other if the GUI will be guaranteed to be written in Python also?

Edit: I have found a lot of IPC libraries, like here: Efficient Python to Python IPC or ActiveMQ or RabbitMQ or ZeroMQ or.

The best looking options I have found so far are:

• rabbitmq
• zeromq
• pyro

Are they appropriate slutions to this problem? If not why? And if something is better, please tell me why also.

Answer 1

The three you mentioned seem a good fit and will uphold your requirements. I think you should go on with what you feel most comfortable\familiar with.

From my personal experience, I do believe ZeroMQ is the best combination between efficiency, ease of use and inter-operability. I had an easy time integrating zmq 2.2 with Python 2.7, so that would be my personal favorite. However as I said I'm quite sure you can't go wrong with all 3 frameworks.

Half related: Requirements tend to change with time, you may decide to switch framework later on, therefore encapsulating the dependency on the framework would be a good design pattern to use. (e.g. having a single conduit module that interacts with the framework and have its API use your internal datastructures and domain language)

Answer 2

I've used the Redis engine for this. Extremely simple, and lightweight.

Server side does:

import redis
r = redis.Redis() # Init
r.subscribe(['mychannel']) # Subscribe to "channel"
for x in r.listen():
  print "I got message",x

Client side does:

import redis
r = redis.Redis() # Init
r.publish('mychannel',mymessage)

"messages" are strings (of any size). If you need to pass complex data structures, I like to use json.loads and json.dumps to convert between python dicts/arrays and strings - "pickle" is perhaps the better way to do this for python-to-python communication, though JSON means "the other side" can be written in anything.

Now there are a billion other things Redis is good for - and they all inherently are just as simple.

Answer 3

You are asking for a lot of things from the framework; network enabled, multi-platform, multi-language, high performance (which ideally should be further specified - what does it mean, bandwidth? latency? what is "good enough"; are we talking kB/s, MB/s, GB/s? 1 ms or 1000 ms round-trip?) Plus there are a lot of things not mentioned which can easily come into play, e.g. do you need authentication or encryption? Some frameworks give you such functionality, others rely on implementing that part of the puzzle yourself.

There probably exists no silver bullet product which is going to give you an ideal solution which optimizes all those requirements at the same time. As for the 'additional' component of your question - yes, if you restrict language requirements to python only, or further distinguish between key vs. nice-to-have requirements, there would be more solutions available.

One technology you might want to have a look at is Versile Python (full disclosure: I am one of the developers). It is multi-platform and supports python v2.6+/v3, and java SE6+. Regarding performance, it depends on what are your requirements. If you have any questions about the technology, just ask on the forum.