error while starting kafka broker

Question

I was able to successfully set up zookeeper and one kafka broker yesterday. Everything worked as expected. I shut down kafka (ctrl + c) and then zookeeper.

Today I started zookeeper and when I started kafka (bin/kafka-server-start.sh config/server0.properties), I get the following error. I tried various remedies suggested (removing completely my kafka installation and doing it again from scratch). Still I get same error.

[2016-09-28 16:15:55,895] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable) java.lang.RuntimeException: A broker is already registered on the path /brokers/ids/0. This probably indicates that you either have configured a brokerid that is already in use, or else you have shutdown this broker and restarted it faster than the zookeeper timeout so it appears to be re-registering.         at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:305)         at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:291)         at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:70)         at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:51)         at kafka.server.KafkaServer.startup(KafkaServer.scala:244)         at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)         at kafka.Kafka$.main(Kafka.scala:67)         at kafka.Kafka.main(Kafka.scala) [2016-09-28 16:15:55,896] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer) 

All set up in mac

Answer 1

I have faced the same issue while setting up the kafka and zookeeper multinode cluster.

Root Cause :

As zookeeper ensemble takes some time to setup so when we start kafka brokers, all zookeeper nodes might have not joined zookeeper ensemble yet.

So It may be the case that one kafka node connected to a zookeeper node which is not yet in the ensemble.

Another reason can be kafka broker is killed instead of stopped. When you kill your kafka process or it crashes and it does not close its zookeeper connection correctly.

Solution :

To avoid this situation make sure your zookeeper ensemble is up and running before starting kafka brokers.

Add proper waits between zookeeper and kafka broker services start and stop.