Distribution of "tasks" to multiple hosts in python

Question

I am wondering whether there's a lightweight way to distribute python "tasks" among different hosts. Here's what I mean:

Master script running in e.g. London Client 1 running in London Client 2 running in New York Client 3 running in Hong Kong

The clients would have a way (e.g. SOAP) to communicate with the master. The master would send the clients the data and then wait for the responses generated by the clients.

All of this could be achieved easily simply by using SOAP messages.

However, what if the master wants to distribute the code to be run on the clients together with the input data? I am not sure what's the pythonic way of doing this.

Thanks, Matyas

Answer 1

0MQ

I would look at zeromq. However, before I built a system that allowed arbitrary code to be executed, I would make sure I had a thorough understanding of cryptographic authentication!