Is it helpful to use ZeroMQ to build a peer-to-peer workload scheduler?

Question

I am coding a workload scheduler. I would like my piece of software to be a peer-to-peer scheduler, ie. a node only knows some neighbours (other nodes) and use them to reach other nodes.

Each node would have its own weighted-routing table to send messages to other peers (basically based on the number of hops), ie. "I want the master to give me my schedule" or "is resource A available on node B ?" : which neighbor is the closest to my target ?

For instance I have written my own routing protocol using XML-RPC (xmlrpc-c) and std::multimaps / std::maps.

I am thinking of using ZeroMQ to optimze my data streams :

• queueing can reduce the network load between peers ;
• subscriptions can be used to publish upgrades.

As a consequence :

• I would need to open as many sockets as I would create new types of connections ;
• Each node would need to be a client, a server, a publisher, a subscriber, a broker and a directory ;
• I am not sure that my "peer-to-peer architecture" is compatible with the main purpose of ZeroMQ.

Do you think that ZeroMQ can be a helpful concept to use ?

Answer 1

It would be helpful to know exactly what you mean by "routing protocol". That sounds like you mean the business logic of routing to a particular peer. Knowing more fully what you're looking to achieve with ZeroMQ would also be helpful.

Have you read the ZeroMQ Guide? ZeroMQ is a pretty different beast and without spending some time to play with it, you'll likely find yourself confused. As a bonus, reading the guide will also help you answer this question for yourself, since you know your requirements better.

ZeroMQ was designed to build robust distributed and multi-threaded applications. Since distributed applications can often take the form of "peer-to-peer", ZeroMQ could indeed be a good fit for your needs.