master node in multi-node kafka cluster

Question

I am trying to set up a multi node kafka cluster and have the following three config files. But how do I know which node is the master?

broker.id=1
port=9093
log.dir=/data/logs/kafka/kafka-logs-1

in config/server1.properties

broker.id=2
port=9093
log.dir=/data/logs/kafka/kafka-logs-2

in config/server2.properties

broker.id=3    
port=9093
log.dir=/data/logs/kafka/kafka-logs-3

in config/server3.properties

Answer 1

Kafka does not have a notion of "master" and "slave" nodes with regards to its brokers (unlike related technologies such as Apache Hadoop, Spark, or Storm). "All brokers are created equal", so to speak. So this is good news -- you don't need to configure anything special! All you need to do is to start the three broker processes*, and they will transparently figure out how to cooperate.

And yes, there is a notion of leaders and followers for topic partitions/replicas, but all of this is automagically managed behind the scenes for you. Also, leader/follower assignments are dynamic and may change at runtime. For example, if broker B1 is the current leader of partition P8 and broker B1 subsequently crashes, then one of the brokers that are the followers for partition P8 will automatically be elected as the new leader for P8 (say, broker B3). Once broker B1 is alive again, it will typically become a follower of the newly elected leader B3 with regards to partition P8.

*I assume these processes will run on different machines, otherwise (i.e. in case all three broker processes run on a single machine) you must assign different port settings to each process.