I am trying to design a replay mechanism that will enable users to replay messages from the queues. The best design I have come up for an exchange that contains multiple queues and multiple consumers is:
Create a recorder service that will:
Subscriber request for replay.
Questions:
1. Does that make sense?
2. Am I inventing the wheel? Is there a rabbit inherent solution? plugin?
3. Does creating multiple exchanges considered a good practice?
In this solutionan exchange for each queue is created in order to publish the same message.
Another solution:
1. Create an additional queue "ReplayQueue" for each queue. set a TTL (let's say a month).
2. Each time a user requests a replay let him replay from its own ReplayQueue without acking.
This solution is a bit problematic because.
Messages in RabbitMQ can not be replayed, since they are removed once they are ack:ed. However, a RabbitMQ client can nack (negative acknowledgement) a message when it fails to handle the message; this is useful in case of a temporary failure on the consumer side. The message will simply be added back to the queue.
By default, RabbitMQ will send each message to the next consumer, in sequence. On average every consumer will get the same number of messages. This way of distributing messages is called round-robin.
This pattern is commonly known as Remote Procedure Call or RPC. In this tutorial we're going to use RabbitMQ to build an RPC system: a client and a scalable RPC server. As we don't have any time-consuming tasks that are worth distributing, we're going to create a dummy RPC service that returns Fibonacci numbers.
In order to consume messages there has to be a queue. When a new consumer is added, assuming there are already messages ready in the queue, deliveries will start immediately. The target queue can be empty at the time of consumer registration. In that case first deliveries will happen when new messages are enqueued.
- Does that make sense?
Yes
- Am I inventing the wheel? Is there a rabbit inherent solution? plugin?
You are not reinventing the wheel. There is AFAIK no rabbit solution and no out of the box solution.
Your first solution is in my opinion good. The Another solution
is very problematic, because a healthy rabbit is an empty one and rabbit is not a datastore.
You will have a queue (STORE
) where all published messages should be routed to. Instead of binding STORE
with all the binding keys, you could consider using a topic exchange. At the price that binding key must not contain . # *
and a slight overhead when the message is routed. The STORE
queue will bind with the binding key #
.
You could have a look at firehose.
Why a web request? You can use Rabbit with an RPC call:
reply-to
queue name. The queue may be exclusive to the client and request.You could also have a look at the direct-reply-to pattern.
- Does creating multiple exchanges considered a good practice?
Yes, as soon as you need it. For your specific case, in my opinion an exchange per subscriber is not necessary. The request contains already the queue name. You could simply publish to this queue using the default exchange amq.direct
with the routing key equal to the queue name. If you want an exchange I would create an unique exchange (eg. "replay") and have each subscriber bind their replay queues to this exchange. The "replay" exchange can be a convention or sent with the request.
The first solution is feasible. Given that rabbit MQ ships with a tracing plugin
, storing the message in DB becomes even more easier as it simply boils down to consuming from a queue bound to amq.rabbitmq.trace
exchange.
This exchange is specific to a vhost
and every vhost has its own amq.rabbitmq.trace
exchange. In addition when creating a new trace it is possible to restrict on which queue/exchange to enable tracing , therby giving the flexibility to disable tracing where it is not required.
Refer the following links to configure tracing:
https://www.rabbitmq.com/firehose.html
https://www.rabbitmq.com/blog/2011/09/09/rabbitmq-tracing-a-ui-for-the-firehose/
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With