How to limit throughput with RabbitMQ?

Question

Where did the question:

We are using RabbitMQ as task queue. One of specific tasks - sending notices to Vkontakte social network. They api has limit to request per seconds and this limit based on your application size. Just 3 calls for app with less then 100k people and so on. So we need to artificially limit out request to they service. Now this logic is application based. It simple while you can use just one worker per such queue, just set something like sleep(300ms) and be calm. But when your should use N workers this synchronization becomes not trivial.

How to limit throughput with RabbitMQ?

Based on story above. If it were possible set prefetch size not only message based but time based to this logic can be much simple. For example, "qos to 1 message per fetch not faster then 1 time in seconds" or so on.

1. Is there something like this?
2. May be other strategy about this?

Answer 1

This is not possible out of the box with RabbitMQ.

You're right, with distributed consumers this throttling becomes a difficult exercise. I would suggest having a look at ZooKeeper which would allow you to synchronize all consumers and throttle processing of messages by leveraging it's Znodes / Watches for throttled yet scalable solution.