Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

multi-threading based RabbitMQ consumer

We have a windows service which listen to single RabbitMQ queue and process the message.

We would like to extend same windows services so that it can listen to multiple queues of RabbitMQ and process the message.

Not sure if that can be possible by using multi-threading as each thread will have to listening (blocking) the queue.

As I am very new to multi-threading, need high level guideline on following point, which will help me to start building the prototype.

  1. Is it possible to listen multiple queues in single application by using threading?
  2. How to handle the situation where if any single thread got shut down (due to exception etc.), how to bring back without restarting the whole windows services.
  3. Any design pattern or open source implementation which can help me to handle this kind of situation.
like image 902
Mahesh Avatar asked Feb 18 '14 01:02

Mahesh


People also ask

Is RabbitMQ multithreaded?

Use multiple queues and consumers Queues are single-threaded in RabbitMQ, and one queue can handle up to about 50 thousand messages.

Does RabbitMQ support multiple consumers?

RabbitMQ has a plugin for consistent hash exchange. Using that exchange, and one consumer per queue, we can achieve message order with multiple consumers. The hash exchange distributes routing keys among queues, instead of messages among queues. This means all messages with the same routing key will go the same queue.

Is RabbitMQ channel thread safe?

In Rabbitmq, Channel is not thread-safe. So it's better to use different threads to handle different channels.

How consumer Works in RabbitMQ?

In order to consume messages there has to be a queue. When a new consumer is added, assuming there are already messages ready in the queue, deliveries will start immediately. The target queue can be empty at the time of consumer registration. In that case first deliveries will happen when new messages are enqueued.


1 Answers

I like how you wrote your question - it started out very broad and focused in to specifics. I have successfully implemented something very similar, and am currently working on an open-source project to take my lessons learned and give them back to the community. Unfortunately, though- I have yet to package my code up neatly, which doesn't help you much! Anyway, to answer your questions:

1. Is it possible to use threading for multiple queues.

A: Yes, but it can be full of pitfalls. Namely, the RabbitMQ .NET library is not the best-written piece of code out there, and I have found it to be a relatively cumbersome implementation of the AMQP protocol. One of the most pernicious caveats is how it handles the "receiving" or "consuming" behavior, which can cause deadlocks quite easily if you aren't careful. Fortunately, it is well-illustrated in the API documentation. Advice - if you can, use a singleton connection object. Then, in each thread, use the connection to create a new IModel and corresponding consumers.

2. How to gracefully handle exceptions in threads - I believe this is another topic and I will not address it here as there are several methods you can use.

3. Any open-source projects? - I liked the thinking behind EasyNetQ, although I ended up rolling my own anyway. I'll hopefully remember to follow back when my open source project is completed, as I believe it is an even better improvement than EasyNetQ.

like image 93
theMayer Avatar answered Oct 12 '22 02:10

theMayer