In Kafka java api, what does the key and value of the concumerRecord mean?

Question

I am now studying about Kafka java api. Among contents about concumer, I often saw ConsumerRecords. I want to know exactly what the key and value mean. I already read official doc, but i cannot find answer. please, share your wisdom.

Answer 1

You can think of each record in a Kafka topic as a key-value pair. The key, by default, is used for routing a message to a particular partition of that topic. Let's take an example. Let's say we have a topic cars with 4 partitions. Now, we push the following messages (K, V pairs) to this topic.

1. (sedan, fusion)
2. (suv, highlander)
3. (truck, Freightliner)
4. (bike, harley)

What Kafka will do for each of these messages, is compute a hash of the key modulo number of partitions. So, message-1 will go to partition no.hashCode(sedan) % 4 of cars. Message-2 will go to partition no. hashCode(suv) % 4, and so on. If you don't specify any key, i.e. key is null, then depending on the default partitioning strategy (e.g. round robin) Kafka will assign a partition to the message. In that case messages will looks like:

1. (null, fusion)
2. (null, highlander)
3. (null, Freightliner)
4. (null, harley)

Now, messages 1, 2, 3 and 4 will be assigned to partition no. 0, 1, 2 and 3. If you specify a key, and you want to override the default partitioning strategy with your own, Kafka allows you to do that too.