In Kafka is each message replicated across all partitions of a topic?

Question

If a topic has 4 partitions, and a publisher sends a message to the topic, will that same message be replicate across all four partitions or only one?

Answer 1

Partitioning and replication are two different things.

Partitioning is for scalability. A topic is partitioned in one or more partitions distributed on different brokers so that more consumers can connect to these brokers in order to receive messages sent to the same topic but from different partitions. Increasing partitions increases scalability and the possibility to have more consumers to get messages from the same topic. Answering your question, each message sent to a topic comes into only one partition (of the topic itself).

Replication is for fault-tolerance. You can specify a replication factor on topic creation and it means that every partition for that topic is replicated more times on different brokers. One replica is the "leader" where producer sends and consumer gets messages; other replicas are "follower" which have copies of messages from the "leader" replica. If the broker which handles the "leader" replica goes down, one of the "follower" becomes leader.

Answer 2

Replication does not occur across partitions. Each message goes into a single partition of the topic, no matter how many partitions the topic has.

If you have set the replication-factor for topic to a number larger than 1 (assuming you have multiple brokers running in the cluster), then each partition of the topic is replicated across those brokers.