Can someone explain what message brokers are used for?

Question

In my line of work it's hard to go five minutes without someone extolling the virtues of MQ Series or MSMQ or the like, and I always wonder, after the sparkle of buzzwords has passed, what are some actual examples of these wonderful devices out in the real world.

What I'm looking for is something that might inspire me to find a use for one of these or give me some kind of metric I can use to evaluate a message bus/message broker/message queue -- hell, even something that will explain what the differences are between the aforementioned message* things.

Answer 1

Message queuing solutions like MQ Series or MSMQ are used extensively for integrating distributed enterprise applications, especially running on different platforms. Done right (with persistent queues, asynchronous design rather than 'RPC over MQ' and attention to operational requirements), this gives you high availability compared to synchronous request/reply integrations such as RPC or boilerplate web services (whose availability is the product of the respective availabilities: integrating ten systems with 99 % availability synchronously gives you a combined availability of no more than 90 % -- or worse if there's just one weak link). Mind you, this requires the message queues to have high availability in themselves: we use our mainframe for that purpose (take a guess at which product we're using!).

Message (or integration) brokers and 'buses' are a more complex kettle of fish. They can add

• translation between different content representations (text encodings and code pages)
• supervision, detecting when a target system does not pick up queued messages and raising alarms or automatically restarting
• transformation, when systems do not 'speak the same language' and represent e.g. customer or product records differently: this can in principle help you deploy new versions at different rates
• routing (up to and including publish/subscribe) to decouple a sending system from knowledge of the details of recipients, thus reducing impact of changes in target systems
• orchestration, where you can coordinate messages between a number of systems to track a longer real-life business process (say, from customer order to delivery to invoicing).

I've listed these functionalities in roughly increasing order of difficulty (and potential reward). The higher you get, the more mature your organization (including the business side) needs to be in order to gain the advantages.

Answer 2

Without getting into specifics about particular products, I can give you some of the benefits to using a typical message queuing system.

They are typically good infrastructure for simulating the publisher/subscriber pattern. Think of a big event system where you are not limited to one executable, or even one machine. You put information into these queues, such that the data can be picked up by any application who is listening for it.

Most message queuing systems allow for persistent queues. Think of a typical event system. If the listener is disconnected or otherwise unresponsive at the time of the event, then the event is missed. With a persistent message queue, the message will remain in queue until the listener is reconnected. No data/events are lost in this way.

I don't know about the products you listed, but I know with JMS you can have fine grained control over threading as messages are popped from the queue. In theory, you could have a thread per queue, performing actions in that thread, on the messages as they are pulled off. Alternatively, you have the ability to pull messages from multiple queues and perform actions on them, all within a shared thread.

Answer 3

While I had a very bitter experience with MQ Series partially due to the fact that it was pushed on to us (a Microsoft shop) by the partnering company, the use of MQ Series (Or any Messaging system) was an integral part to the application.

Essentially we were building a process that handled supply chain fullfilment for backorder items. If our partner a distributor didn't have the items their customers wanted, they would send a message into a B2B site, that would target potential companies that could fulfil the order.

We had built two different flavors of integration. The first was an ftp approach where fixed width files were sent back and forth at regular intervals, and we had added all sorts of rules to help ensure we didn't miss any data.

The second was using MQ Series where the messages were placed into a queue using guarenteed delivery. Then we would pop the queue and process the messages. The queing system was great benefit here as it allowed us a reliable way to transmit critical messages that resulted in real money being moved around.

On the flip side with the same MQ Series we had to implement a synchronous query to get information. We wanted it to be synchronous because our users accessing this via the web would wait to get the information. Doing this over MQ Series was a very interesting and painful challenge. The only reason MQ was used here was because it was an existing line of communication and the query functionality already existed.

A second example and this time was using MSMQ was a site that collected information from dialhome code injected into client applications. The dialhome code would collect feature usage statistics like Microsoft's SQM program. When the messages came in to the web service we would drop them on a queue, Then we could have any number of application servers popping the messages and pushing them to the database to be rolled into the warehouse.

MSMQ here ensured we could handle bursts of messages by quickly placing them on the queue. This help the scalability and reliability of the system.

Answer 4

A message queue is useful to implement load balancing. For example, the server receives "job" messages (orders, status messages ...) and distributes them to all listening clients.

The message queue guarantees that a message will be delivered to exactly one client.

If the clients run on different computers, the total load will be distributed and it will be easy to throw another client to the message load when neccessary, the client just has to connect to the queue and will receive (some of) the messages.

Answer 5

A good queuing system makes it easier to do distributed computing over multiple threads, processors, machines, (& even organizations).

A while back (10 years) I used a message sending metaphor to implement a front-office options pricing system for an interdealer broker. We had services implemented in C++, VB6 and Excel/VBA (even using the Excel solver!!), data storage as flat files and sql, end-user applications writen in Excel and VB6, with a complex data model (market data, yield curves and vol surfaces). Asynchronous messaging (with persistent / reliable messages and pub / sub) made the whole thing work very effectively and scalably - we were able to add Tokyo and NY offices to the London infrastructure without even visiting the remote site - just a bog standard install.

I'm a big fan though I am surprised at how far they haven't come in the last 10 years or so.