Advantages & Disadvantages of Brokered vs non-brokered Messaging Systems

Question

I'm trying to design a real-time monitoring & control system that's modular, so it can distributed, and expanded/reconfigured for different hardware & networks.

I've quickly come to the conclusion I'll need some kind of distributed enterprise messaging system. But there are many options out there, each with advantages and disadvantages, and some of them dictate different architectures. I'm trying to work out whether I need a broker or brokerless system, whether I need the message reliability of some systems (e.g. RabbitMQ) or the light-weight high-throughput of a system like ZeroMQ, or the "arrive in order" high throughput of Kafka.

First, do these architectures make sense?

---

ZeroMQ type "Brokerless" system:

Notes:

There can be many "Part A" to each "Part B", and many "Part B" feeding into a "Part C"

Advantages:

• High throughput, low latency
• Easily integrated into components, lightweight deployment (no need to deploy a broker).

Disadvantages

• Messages not guaranteed delivery. Some may be dropped. This may be a problem in the orange highlighted areas. It's not critical for the GUI, but if the local control module is making decisions, it might need all the information. (Thinking about it, just the latest is probably good enough - no point making a decision with out of date data). Similarly, if the network between A and B goes down, the historian will have incomplete history. How critical is this though?
• No "discovery". Relationship between components needs to be more managed.

---

RabbitMQ type Broker system:

Advantages:

• Messages guaranteed delivery.
• Discovery managed through brokers.

Disadvantages

• Much slower, high latency
• More to deploy & maintain (brokers/RabbitMQ need installing on machines, it's not just built into the modules)

  ---

Inbetween options:

I've looked at Kafka. It's brokered, so discovery is taken care of. However, it seems much more lightweight than RabbitMQ and while it doesn't guarantee delivery (thus is faster/lower latency) it does maintain order, which RabbitMQ doesn't. It also buffers messages - so they can be retrieved if there's a network problem.

After writing this down, I'm not sure how important guaranteed delivery is. If the control module gets a message, if it's "old" it doesn't matter. It would be great if the historian had a full history - but is it essential?

It might be an option to implement my own "Message buffer" in ZeroMQ for network communication that stores messages in case of failure. I'd have more control than RabbitMQ, and can just implement it when I need it for messaging over the more unreliable (over the network).

Obviously, weighing up these advantages or disadvantages is my job. My question is: Is there anything else to consider? and Does the architecture for these two options make sense?

I'm planning on most implementation to be in C#, and I currently have zero experience in messaging systems.

Answer 1

Reliability can mean different things. This link from zmq is probably one of the best I have read. But here's a brief explanation of what reliability in the event of hardware failures

Apache Kafka - Message Delivery Guarantee can mean different things. See Message Delivery Semantics. It is important to note that "Kafka's semantics are straight-forward. When publishing a message we have a notion of the message being "committed" to the log. Once a published message is committed it will not be lost as long as one broker that replicates the partition to which this message was written remains "alive". "

RabbitMQ offers some options as well. Read about Clustering and HA. But I personally think that Apache Kafka is inherently (by design) a distributed, partitioned, replicated commit log service and hence solves this problem in a much cleaner manner.

ZMQ I don't know enough about zmq to make an informed conclusion. But I think zmq doesn't attempt to solve the problem of reliability. Instead it is an embeddable networking library which provides a base for performant, scalable clustered applications to interact with each other via messages. However, from what I can tell, it doesn't particularly address the problem of reliably persisting messages (as a broker). Apache Kafka seems to fill this niche very well - it is performance is great, yet offers options of achieving reliability.

Conclusion: I think reliability is not just the responsibility of the broker. Instead, it is the collective responsibility of all the pieces that make up your application. Reliability, performance and scalability can only be achieved by good design and use of the right technologies.

Answer 2

Real-time and guaranteed message delivery are not really possible. If a system really needs real-time data (e.g a stock trading algorithm) then it cares more about getting the very latest price with the lowest latencies over high latency delivery guarantees.

I think you should look at your system and break it into components which:

• Need to be realtime (real time controls, decision making)
• Need to be reliable (historical databases)

Looking at you diagram I think you have a good requirement for two message systems

• zeromq for the realtime control parts
• kafka for the guaranteed delivery historical/database part.

BTW the zmq discovery is quite easily solved with a couple of redundant zmq proxies and some form of DNS.

Answer 3

Your suggestion to "implement your own Message buffer in ZeroMQ for network communication that stores messages in case of failure" seems a viable approach. Did you ever pursue that? I'd be interested in your experiences doing so.

A kind of 'packaged' message pipeline with durability, low latency and high throughput seems ideal. Building it on top of ZMQ gives you much less processing overhead and much less administration/setup headaches.