Local vs. remote queues in pub/sub messaging

Question

If I was building a system with dozens of publishers/subscribers using message queues, it seems I have a few network configuration options:

1. I could have one clustered broker that all machines use - each machine would not have a local queue
2. I could install brokers locally on every machine, and use store-and-forward to deliver messages to remote machines

Different technologies seem to enforce different configurations - e.g., MSMQ requires every machine to have its own local queue, while Tibco EMS seems to often be used in clusters, without local queues on each consumer.

What are the downsides to not having a local queue, and what factors influence the decision?

Answer 1

Not having a local queue that provides a durable message store means that you cannot guarantee message delivery. Using something like RabbitMQ in a cluster with a broker instance locally gives you a durable mechanism to store messages for delivery. If you have to connect over a network connection to a remote broker to send a durable message, you're at a higher risk of network failure.

MSMQ is also store-and-forward, but it doesn't provide any clustered routing capabilities. This means that the application has to do the work (or have a layer on top of it, such as MassTransit or NServiceBus do it for you).

When I think of TIBCO, I think of a centralized cluster of EMS servers that application servers communicate with instead of running a broker instance locally. The GUI tools that wrap around EMS and the BusinessWorks application servers really force a model in that world.

In any of the distributed cases where messages are stored locally, it's important to make sure the machine itself is properly equipped for message storage, with fast disk and sufficient disk for the expected message backlog/capacity.

Answer 2

I would venture the local queue would be necessary in most scenarios.

The local queue is necessary if the message needs to be durable. In other words, if the connection to a remote queue is unreliable and you want the message to eventually reach its subscribers, you would want the store-and-forward capability of the local queue.

If the message does not need to be durable (can be lost after the firing of the event), then the remote shared queue would be an option.

You may want to look at the NServiceBus distributor model, which would be a hybrid of your two scenarios: a local queue on each machine to forward to a remote broker/distributor cluster.