Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to send message to a particular partition in Kafka?

I have created a topic that has many partitions. Using the console producer I want to send messages to particular partitions and view the through the console consumer. On the console producer I have tried this,

kafka-console-producer.bat --broker-list localhost:9092 --topic sample  --property parse.key=true --property key.separator=,

Send messages as,

key1,another-message

But I am just confused on whether key1 represents partition number.

Using the console consumer I viewed the messages,

kafka-console-consumer.bat --zookeeper localhost:2181 --topic sample

I want to view the messages according to the partitions. Is this the right way to view the messages on the console consumer? Can anyone please provide a clear understanding on this?

like image 240
AmZ62210 Avatar asked May 14 '18 06:05

AmZ62210


People also ask

Can I send a set of messages to a specific Kafka partition?

As seen in previous examples, when we send messages ( ProducerRecord ) we can specify key and value. While sending messages, if partition is not explicitly specified, then keys can be used to decide to which partition message will go. All messages with the same key will go to the same partition.

How do you send a message to a specific partition in Kafka Python?

The key is not the partition number but Kafka uses the key to specify the target partition. The default strategy is to choose a partition based on a hash of the key or use round-robin algorithm if the key is null. If you need a custom algorithm to map the messages to partitions, you need to implement org. apache.

How is data assigned to a specific partition in Kafka?

The message gets assigned a partition based on the key and all messages for the same key ends at the same partition. On the consumer, you subscribe to the whole topic (without explicitly asking for a partition) and Kafka will handle the distribution of partitions between all the consumers available.


1 Answers

You can specify partition number directly in the ProducerRecord, but not with kafka-console-producer.

The key is not the partition number but Kafka uses the key to specify the target partition. The default strategy is to choose a partition based on a hash of the key or use round-robin algorithm if the key is null.

If you need a custom algorithm to map the messages to partitions, you need to implement org.apache.kafka.clients.producer.Partitioner interface. The name of you class must be set as a partitioner.class property of the producer.

like image 96
Katya Gorshkova Avatar answered Sep 28 '22 09:09

Katya Gorshkova