Kafka Broker vs Topic

Question

A topic usually resides within the broker (server process) and has multiple partitions, but can it span across multiple brokers or nodes (machines)?

i.e, lets say a topic is divided into two partitions, partition-1 and partition-2. Can partition-1 belong to broker-1 and partition-2 of the same topic belong to a different broker-2 or a different node altogether?

Answer 1

A Kafka cluster consists of one or more servers (Kafka brokers).

Each Broker can have one or more Topics. Kafka topics are divided into a number of partitions, each partition can be placed on a single or separate machine to allow for multiple consumers to read from a topic in parallel.

Answer 2

Yes, different partitions of a topic can be stored in multiple machines. But, a single partition has to be stored entirely in one machine.

Answer 3

Kafka Broker as the name stated its kind of controller or server who receives the request and does the particular operation. For example, Kafka broker receives messages from producers and stores them on disk keyed by unique offset. A broker can response unique message from topic, partition, and offset to the consumer on request

Whenever we think about Kafka Broker it’s not a single unit but a group of the server which shares information using ZooKeeper. Kafka Brokers do not sit on one machine but able to distribute across a distributed cluster environment. This group of brokers deployed on Cluster coordinated by Zookeeper can be visualized Kafka Cluster. Brokers are the heart of Kafka Cluster and its connector to the outer words such as consumer, producer, confluent connector.

Kafka Topic consists of multiple partitions distributed across Brokers. One partition can be treated as leader other partitions will be a follower and in other broker and message get replicated using ISR.

You can assign leader partition by using Reassign Partitions Tool you can refer below link for more detail

https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools

Answer 4

Th kafka topics are replicated across multiple brokers so that multiple consumers can receive the data parallely or in case the one broker goes down , the data can be retrieved from other broker with the same topic partition.