How Kafka Nodes and zookeeper will communicate with each other?

Question

I could not find any details on communication between Nodes and Zookeeper . suppose more Garbage collection is happening in Kafka nodes , What would be result of it ?

1. Will Zookeeper disconnect the communication with respective node ?
2. If Zookeeper will disconnect the respective node what would be result ?

Answer 1

Apache Kafka uses Zookeeper to select a controller, to maintain cluster membership and to store configuration, including the list of topics in the cluster.

In order to remain part of the Kafka cluster, each broker has to send keep-alive to Zookeeper in regular intervals. This is something every Zookeeper client does by default. If the broker doesn't heartbeat Zookeeper every zookeeper.session.timeout.ms milliseconds (6000 by default), Zookeeper will assume the broker is dead. This will cause leader election for all partitions that had a leader on that broker. If this broker happened to be the controller, you will also see a new controller elected.

So, if garbage collection pause takes longer than 6000 milliseconds, you will see the broker disconnecting from Zookeeper and a bunch of leader elections as a result. Since garbage collection pressure rarely results in just one long pause, you'll probably experience what we call "flapping" - brokers will keep disconnecting and reconnecting to Zookeeper, lots of leader elections and lots of ISR shrink/expand events.

The reverse is also true: If you see lots of broker "flapping", GC logs are a good place to start looking.