Menu
We are developing a kafka based streaming system in which the producer would produce to multiple partitions within its topic and a single consumer would consume from the topic. I know that kafka maintains message order within partitions, but can we maintain a global message order between partitions within a topic?
231
First of all, Kafka only guarantees message ordering within a partition, not across partitions. This places a burden on the producers and consumers to follow certain Kafka design patterns to ensure ordering. For example, the ability to partition data by key and one consumer per partition.
When the number of consumers is lower than partitions, same consumers are going to read messages from more than one partition. In your scenario, a single consumer is going to read from all your partitions. This type of consumer is known as exclusive consumer. This happens when consumer groups have only one consumer.
As per Kafka guarantees: Messages sent by a producer to a particular topic partition will be appended in the order they are sent. To ensure strict ordering across all events, the topic should be single partitioned.
Short answer: no, Kafka does not provide any ordering guarantees between partitions.
Long answer: I don't quite understand your problem. If you are saying you have only one consumer consuming your topic, why would you have more than 1 partition in that topic and reinvent the wheel trying to maintain order between partitions? If you want to leave some space for future growth, e.g. adding another consumer to consume a part of partitions, then you'll have to rethink your "global message order" idea.
Do you really need ALL messages to be processed in order? Or maybe you could partition by client/application/whatever and maintain order per partition? In most cases you don't really need that global message order, but just have to partition your data properly.
Maintaining order between multiple consumers is a really tough problem to solve, and even if solved correctly you'll just neglect all Kafka benefits.
71
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
