AMQP: acknowledgement and prefetching

Question

I try to understand some aspects of AMQP protocol. Currently I have project with RabbitMQ and use python pika library. So question is about acknowledgement and message prefetching.

1. Consider we have a queue with only consumer (for sure this queue was declared as exclusive). So do I understand correctly: no matter if I consume with or with no ack flag? Anyway I should not be able to process several messages simultaneously and there are no another consumers that could take some of other still queued messages. Even better not to turn acknowledgement on, because possibly this may reduce AMQP server load.

2. Prefetch count doesn't mean anything if there is no acknowledgement. Correct?

3. I am not sure how prefetching works. I have a callback on new message and in its finally statement I acknowledge or reject the message. This is the only function and no matter how big prefetch count would be - anyway another message would not be processed until the current one is completed. So why should I need change prefetch_count value?

thanks in advance.

Answer 1

With autoack flag unset, if your application failed during message processing all received messages will be lost. If such situation is quite rare and message lose is appropriate option in your application (for example, but no limited to, logs processing) you may turn autoack off.

And yes, having autoack unset requires simpler broker logic, so it utilize less resources.

As to prefetch count (and prefetch size), these options tell broker how large maybe the payload that sent to client in advance to single message. Usually it is used to save time on network operations to wait for a new messages. When prefetch size used client will receive one or more messages that has total size equal or less to preset pretch size (and/or count, which is less).

Both prefetch count and prefetch size rules is they set are applied. When one of them is set to zero (unset) it will not be applied.

The most important thing, prefetching defines behavior to send messages in advance of more message that client has unacked.

In conjunction these two settings produce something like this:

Prefetch with messages count limit and sending messages in advance:

Conditions:

• Queue: N messages x 1kb
• Prefetch: prefetch-size=5kb, prefetch-count=4
• Autoack: off

The workflow:

• Broker sends 4 messages (limited by prefetch-count=4) to client. 4 messages will be marked as un-acked and moved out from queue (so they wont be delivered to other clients).
• Client ack 1 message.
• Broker has -1 message unacked (remove that message) and send 1 more message to client (+1 un-aked, -1 from queue, while client already has 3 messages un-acked).
• Client ack the rest 3 messages + new delivered one.
• Broker has -4 messages un-acked and send 4 message again, +4 un-aked, -4 from queue.
• Client ack 1 message and fails.
• Broker will -1 un-acked and then move the rest un-acked to queue back, so -3 un-aked and +3 queue, so they may be delivered to this or other client again.

Prefetch with large messages:

Conditions:

• Queue: 1 message x 5Kb, N messages x 1kb
• Prefetch: prefetch-size=5kb, prefetch-count=2
• Autoack: off

The workflow:

• Broker sends 1 message (limited by prefetch-size=5kb) to client and that message marked as un-aked.
• Client ack 1 message.
• Broker has -1 messages un-acked, send 2 messages again (limited by prefetch-count=2, note, that only first message was 5kb, the rest are 1kb) and these messages marked as un-acked.
• Client ack 1 message and fails.
• Broker will move acked message out of messages queue and the rest of un-acked messages will be moved again to queue they belongs to, so they may be delivered to this or other client again.

With auto-ack:

Conditions:

• Queue: N messages x 1kb
• Prefetch: prefetch-size=5kb, prefetch-count=10
• Autoack: on

The workflow:

• While both prefetch-size and prefetch-count ignored when no-ack set to true (that is how auto-ack functionality called in RabbitMQ and in AMQP docs), messages will be sent to client one-by-one and removed from queue after successful sending.

Note, that AMQP has asynchronous architecture, so under certain conditions two clients MAY receive single message at the same time. Also un-acked message MAY be delivered to the same client back (especially if you have single client).

Also, look at prefetch-size and prefetch-count official documentation and experiment a bit with these options.

P.S.: autoack is basically no-ack AMQP flag set to true.