How to guarantee order in Kafka partition

Question

Ok so I understand that you only get order guarantee per partition.

Just random thought/question.

Assuming that the partition strategy is correct and the messages are grouped correctly to the proper partition (or even say we are using 1 partition)

I suppose that the producing application must send each message 1 by 1 to kafka and make sure that each message has been acked before sending the next one right?

Answer 1

Yes, you are correct that the order the producing application sends the message dictates the order they are stored in the partition.

Messages sent by a producer to a particular topic partition will be appended in the order they are sent. That is, if a message M1 is sent by the same producer as a message M2, and M1 is sent first, then M1 will have a lower offset than M2 and appear earlier in the log. http://kafka.apache.org/documentation.html#intro_guarantees

However, if you have multiple messages in flight simultaneously I am not sure how order is determined.

You might want to think about the acks config for your producer as well. There are failure conditions where a message may be missing if the leader goes down after M1 is published and a new leader receives M2. In this case you won't have an out of order condition, but a missing message so it's slightly orthogonal to your original question but something to consider if message guarantees and order are critical to your application. http://kafka.apache.org/documentation.html#producerconfigs

Overall, designing a system where small differences in order are not that important can really simplify things.

Answer 2

sync send message one by one（definitely slow!）,
or async send message in batch with max.in.flight.requests.per.connection = 1