Partition selection in Kafka

Question

I am curious about that If I have topic A and B which they have same number of partitions, if I send message with key x to topic A it goes partition 0 let's say. When I use exactly the same key for topic B but they are independent, in topic B for key x, does the message still go to partition on topic B during kafka streams process ?

Answer 1

By default, Kafka makes use of the DefaultPartitioner (org.apache.kafka.clients.producer.internals.DefaultPartitioner) in order to distribute messages across topic partitions:

/**
 * Compute the partition for the given record.
 *
 * @param topic The topic name
 * @param key The key to partition on (or null if no key)
 * @param keyBytes serialized key to partition on (or null if no key)
 * @param value The value to partition on or null
 * @param valueBytes serialized value to partition on or null
 * @param cluster The current cluster metadata
 */
public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
    if (keyBytes == null) {
        return stickyPartitionCache.partition(topic, cluster);
    } 
    List<PartitionInfo> partitions = cluster.partitionsForTopic(topic);
    int numPartitions = partitions.size();
    // hash the keyBytes to choose a partition
    return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;
}

Essentially, the DefaultPartitioner makes use of MurmurHash, a non-cryptographic hash function which is usually used for hash-based lookup. This hash is then used in a modulo operation (% numPartitions) in order to ensure that the returned partition is within the range [0, N] where N is the number of partitions of the topic.

Coming to your question now, as far as the serialized key to partition on is the same and not null then both messages will be placed to the same partition (assuming that both topics have the same partitioning).