Demultiplexing messages from a queue to process in parallel streams using amqp?

Question

I am trying to figure out if I can switch from a blocking scenario to a more reactive pattern.

I have incoming update commands arriving in a queue, and I need to handle them in order, but only those regarding the same entity. In essence, I can create as many parallel streams of update events as I wish, as long as no two streams contain events regarding the same entity.

I was thinking that the consumer of the primary queue would possibly be able to leverage amqp's routing mechanisms, and temporary queues, by creating temporary queues for each entity id, and hooking a consumer to them. Once the subscriber is finished and no other events regarding the entity in question are currently in the queue, the queue could be disposed of.

Is this scenario something that is used regularly? Is there a better way to achieve this? In our current system we use a named lock based on the id to prevent concurrent updates.

Answer 1

There are at least 2 Options:

A single queue for each entity And n Consumers on one Entity-Queue.

One queue with messages of all entities. Where the message contains data what it is for an entity. You could than split this up into several queues (One AMQP-Queue for one type of entity) or by using a BlockingQueue implementation.

Benefits of splitting up the Entities in qmqp-queues

• You could create an ha-setup with rabbitmq
• You could route messages
• You could maybe have more than one consumer of an entity queue if it is necessary someday (scalability)
• Messages could be persistent and therefore recoverable on an application-crash

Benefits of using an internal BlockingQueue implementation

• It is faster (no net-io obviously)
• Everything has to happen in one JVM

Anyway it does depend on what you want since both ways could have their benefits.

UPDATE: I am not sure if I got you now, but let me give you some resources to try some things out. There are special rabbitmq extensions maybe some of them can give you an idea. Take a look at alternate exchanges and exchange to exchange bindings.

Also for basic testing, I am not sure if it covers all rabbitmq features or at all all amqp features but this can sometimes be usefull. Keep in mind the routing key in this visualization is the producer name, you can also find there some examples. Import and Export your configuration.