Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Implementing a message bus using ZeroMQ

I have to develop a message bus for processes to send, receive messages from each other. Currently, we are running on Linux with the view of porting to other platforms later.

For this, I am using ZeroMQ over TCP. The pattern is PUB-SUB with a forwarder. My bus runs as a separate process and all clients connect to SUB port to receive messages and PUB to send messages. Each process subscribes to messages by a unique tag. A send call from a process sends messages to all. A receive call will fetch that process the messages marked with the tag of that process. This is working fine.

Now I need to wrap the ZeroMQ stuff. My clients only need to supply a unique tag. I need to maintain a global list of tags vs. ZeroMQ context and sockets details. When a client say, initialize_comms("name"); the bus needs to check if this name is unique, create ZeroMQ contexts and sockets. Similarly, if a client say receive("name"); the bus needs to fetch messages with that tag.

To summarize the problems I am facing;

  1. Is there anyway to achieve this using facilities provided by ZeroMQ?
  2. Is ZeroMQ the right tool for this, or should I look for something like nanomsg?
  3. Is PUB-SUB with forwarder the right pattern for this?
  4. Or, am I missing something here?
like image 507
fortytwo Avatar asked Jul 07 '14 23:07

fortytwo


People also ask

Is ZeroMQ a message broker?

ZeroMQ (also spelled ØMQ, 0MQ or ZMQ) is an asynchronous messaging library, aimed at use in distributed or concurrent applications. It provides a message queue, but unlike message-oriented middleware, a ZeroMQ system can run without a dedicated message broker; the zero in the name is for zero broker.

What is ZeroMQ used for?

ZeroMQ provides a whole slew of language APIs which run on most operating systems and allows you to communicate seamlessly between all sorts of programs. It also provides a collection of patterns, such as request-reply and publish-subscribe which assist you in creating and structuring your network.

Do people still use ZeroMQ?

zeromq is still very much in wide use. nanomsg doesn't have nearly as much documentation and community support as zeromq does.

What is a ZeroMQ connection?

ZeroMQ (also known as ØMQ, 0MQ, or zmq) looks like an embeddable networking library but acts like a concurrency framework. It gives you sockets that carry atomic messages across various transports like in-process, inter-process, TCP, and multicast.


1 Answers

Answers

  1. Yes, ZeroMQ is capable of serving this need

  2. Yes. ZeroMQ is a right tool ( rather a powerful tool-box of low-latency components ) for this. While nanomsg has a straight primitive for bus, the core distributed logic can be integrated in ZeroMQ framework

  3. Yes & No. PUB-SUB as given above may serve for emulation of the "shout-cast"-to-bus and build on a SUB side-effect of using a subscription key(s). The WHOLE REST of the logic has to be re-thought and designed so as the whole scope of the fabrication meets your plans (ref. below). Also kindly bear in mind, that initial versions of ZeroMQ operated PUB/SUB primitive as "subscription filtering" of the incoming stream of messages being done on receiver side, so massive designs shall check against traffic-volumes / risk-of-flooding / process-inefficiency on the massive scale...

  4. Yes. ZeroMQ is rather a well-tuned foundation of primitive elements ( as far as the architecture is discussed, not the power & performance thereof ) to build more clever, more robust & almost-linearly-scaleable Formal Communication Pattern(s). Do not get stuck to PUB/SUB or PAIR primitives once sketching Architecture. Any design will remain poor if one forgets where the True Powers comes from.

A good place to start a next step forward towards a scaleable & fault-resilient Bus

Thus a best next step one may do is IMHO to get a bit more global view, which may sound complicated for the first few things one tries to code with ZeroMQ, but if you at least jump to the page 265 of the Code Connected, Volume 1, if it were not the case of reading step-by-step thereto.

The fastest-ever learning-curve would be to have first an un-exposed view on the Fig.60 Republishing Updates and Fig.62 HA Clone Server pair for a possible High-availability approach and then go back to the roots, elements and details.

like image 107
user3666197 Avatar answered Nov 04 '22 06:11

user3666197