Kafka message ordering in partition while producer retry

Question

According to producer configs, there are: retries and max.in.flight.requests.per.connection. Suppose that retries > 0 and max.in.flight.requests.per.connection > 1.

Can messages arrive out of order within ONE partition of topic (e.g. if first message has retries, but second message delivered to broker with the first attempt)?

Or do out of order only happen across several partitions of topic, but within partition order is preserved?

Answer 1

If you set retries to more than 0 and max.in.flight.requests.per.connection to more than 1, then yes messages can arrive out of order on the broker even if they are for the same partition.

You can also have duplicates if for example a message is correctly added to the Kafka logs and an error happens when sending the response back to the client.

Since Kafka 0.11, you can use the Idempotent producer to solve these 2 issues. See http://kafka.apache.org/documentation/#semantics