Python: OpenMPI Vs. RabbitMQ

Question

Suppose that one is interested to write a python app where there should be communication between different processes. The communications will be done by sending strings and/or numpy arrays.

What are the considerations to prefer OpenMPI vs. a tool like RabbitMQ?

Answer 1

There is no single correct answer to such question. It all depends on a big number of different factors. For example:

1. What kind of communications do you have? Are you sending large packets or small packets, do you need good bandwidth or low latency?
2. What kind of delivery guarantees do you need?
3. OpenMPI can instantly deliver messages only to a running process, while different MQ solutions can queue messages and allow fancy producer-consumer configurations.
4. What kind of network do you have? If you are running on the localhost, something like ZeroMQ would probably be the fastest. If you are running on the set of hosts, depends on the interconnections available. E.g. OpenMPI can utilize infiniband/mirynet links.
5. What kind of processing are you doing? With MPI all processes are usually started at the same time, do the processing and terminate all at once.