Batch message from a rabbitMQ queue

Question

I have a stream of requests in my RabbitMQ cluster, and multiple consumers handling them. The thing is - each consumer must handle requests in batches for performance reasons. Specifically there is a network IO operation that I can amortize by batching requests.

So, each consumer would like to maximize the number of requests that it can batch, but not add too much latency.

I could potentially start a timer when a consumer receives the first request and keep collecting requests until one of the two things happen - timer expires, or 500 requests have been received.

Is there a better way to achieve this - without blocking each consumer?

Answer 1

in general, the network aspect of "batching messages" is handled at the level of the basic.qos(prefetch-size, prefetch-count) parameters. In this scheme, the broker will send some number of bytes/messages(respectively) beyond the the unacknowledged messages for a consumer, but the client library doles out messages, in process, one at a time to the application.

To maximize the benefit, the appication can withhold basic.ack() for each message, and periodically issue basic.ack(delivery-tag=n, multiple=True) to acknowledge all messages with a delivery tag <= n.