I have created a topic that has many partitions. Using the console producer I want to send messages to particular partitions and view the through the console consumer. On the console producer I have tried this,
kafka-console-producer.bat --broker-list localhost:9092 --topic sample --property parse.key=true --property key.separator=,
Send messages as,
key1,another-message
But I am just confused on whether key1 represents partition number.
Using the console consumer I viewed the messages,
kafka-console-consumer.bat --zookeeper localhost:2181 --topic sample
I want to view the messages according to the partitions. Is this the right way to view the messages on the console consumer? Can anyone please provide a clear understanding on this?
As seen in previous examples, when we send messages ( ProducerRecord ) we can specify key and value. While sending messages, if partition is not explicitly specified, then keys can be used to decide to which partition message will go. All messages with the same key will go to the same partition.
The key is not the partition number but Kafka uses the key to specify the target partition. The default strategy is to choose a partition based on a hash of the key or use round-robin algorithm if the key is null. If you need a custom algorithm to map the messages to partitions, you need to implement org. apache.
The message gets assigned a partition based on the key and all messages for the same key ends at the same partition. On the consumer, you subscribe to the whole topic (without explicitly asking for a partition) and Kafka will handle the distribution of partitions between all the consumers available.
You can specify partition number directly in the ProducerRecord, but not with kafka-console-producer
.
The key is not the partition number but Kafka uses the key to specify the target partition. The default strategy is to choose a partition based on a hash of the key or use round-robin algorithm if the key is null.
If you need a custom algorithm to map the messages to partitions, you need to implement org.apache.kafka.clients.producer.Partitioner
interface. The name of you class must be set as a partitioner.class
property of the producer.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With