How to produce messages to selected partition using kafka-console-producer?

Question

According to the Kafka documentation:

The producer is responsible for choosing which message to assign to which partition within the topic.

How can I send messages to a selected partition using kafka-console-producer.sh?

I would like to specify some sort of 'partition id' at message sending.

Answer 1

By now, the ConsoleProducer seems to support writing keyed messages to the topic. Kafka will use the hash of the key to distribute the message into partitions, at least with the default behaviour.

Currently, the default separator is \t, so entering key[\t]message will distribute it amongst partitions:

key1    a-message

The separator can be changed by providing the key.separator configuration, for example:

kafka-console-producer --broker-list localhost:9092,localhost:9093 \
  --topic mytopic --property key.separator=,

Send messages like this:

key2,another-message

I have tested this with the default tab and a custom seperator successfuly. The messages were distributed to two separate partitions.

Answer 2

According to the current state of things (Kafka>=0.10.0.1), the kafka-console-producer.sh script and the underlying ConsoleProducer java class support sending data with a key, but such support is disabled by default and has to be enabled from the CLI.

Namely, you need to set the property parse.key. Also, if you want to use something different than a tab character, use key.separator as specified in Cedric's answer.

In the end, the command line would be:

kafka-console.producer.sh --broker-list kafka:9092,kafka2:9092 \
    --topic $TOPIC --property parse.key=true --property key.separator=|