Implementing Task Farm messaging pattern with zeromq

Question

I am using zeromq to solve a problem which involves several hundred (potentially thousands) clients request tasks to be carried out. Each client would request for a specific task to be carried out, and the result(s), when completed, whould be returned back to the client that issued that request.

These are the actors that I have identified so far, in the pattern I have come up with:

• Client: this is the actor that requests a unit of work (or 'job') to be carried out
• Controller: this is the actor that loadbalances the 'jobs' accross available engines
• Engine: this is the actor that receives a job request from the controller and publishes the result back to the client.

I still haven't yet worked out how the engine gets the mesage back to the client. I am guessing that one way for this to be implemented using zeromq would be:

Client:
PUSH job messages on one socket to Controller SUBscribe to completed results on PUBlished by Engine, on another socket

Controller:
PULL job messages from client on one socket PUBlish job messages to engines on another socket (clearly, this will be a forwarding device)

Engine:
SUBscribe to job messages on one socket PUBlish result to another socket

It would be most helpful if someone provide a skeleton/snippet which will show the outline of how this pattern may be implemented, using the zeromq framework.

The code snippet can be in C, C++, PHP, Python or C#

[[Edit]]

After reading up on Task Farms (as suggested by akappa). I think this problem can indeed be modelled by a Task Farm. I have modified my original actors accordingly (and changed the title too).

It would still be very useful if someone who is familiar with zeromq, can sketch out a skeleton that would show how I can use the core components to build such a framework.

Answer 1

There are a variety of approaches to this, and IPython.parallel includes two such implementations with ZeroMQ - one simple and pure-zmq, and another that is more elaborate, with the Controller implemented in Python.

We split the Controller into two actors:

1. Hub - an out-of-the-way process that sees all traffic, and keeps track of the state of the cluster, pushing results to a database, etc., notifying clients about engine connect/disconnect, etc.
2. Scheduler - at its core, a simple ROUTER-DEALER device that forwards requests from the client(s) to the engines, and the replies back up.

Looking at just the task-farming part of our topology:

• Scheduler is a 0MQ Queue device, with a ROUTER and DEALER socket, both of which bind.
• Clients have DEALER sockets, connected to the Scheduler's ROUTER
• Engines have ROUTER sockets connected to the Scheduler's DEALER

Which makes use of these two properties:

• DEALERS LRU load-balance requests across peers
• ROUTERs use identity prefixes to send replies back to the peer that made a particular request.

A toy load-balanced task farm with pyzmq, which routes replies back up to the requesting client: https://gist.github.com/1358832

An alternative, where the results go somewhere, but not back up to the requesting client, is the Ventilator-Sink pattern in the 0MQ Guide.