RabbitMq Dynamically Add/Remove Queues/Consumers

Question

I have an application that receives messages of three types from multiple users at same time and I am handling it using 3 queues/consumers and 3 exchanges. My problem is that when there are thousands of messages from one user in queues then other users are waiting.

I am looking for a solution to execute jobs of every user in parallel. I can create queues dynamically for every user but that's not a good solution because there will be hundreds of queues and consumers. How can I remove idle consumers/queues automatically. Can I use celery and redis for this problem in any way?

Answer 1

It is not a good idea to use many queues in RabbitMQ. Queue is a single thread on RabbitMQ server.

Your problem is "thousands of msgs from one user will block other users", I think the solution is use Priority Queue.

Each user has a counter to record how many messages has been sent in a period time, if this value is large, the messages published by the user will have low priority. So if many messages are waiting in the queue, the messages from other user will be first sent to your consumer jobs. In this situation, you should set a appropriate "prefetch" on consumer's channel.