Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Unable to start kafka with zookeeper (kafka.common.InconsistentClusterIdException)

Tags:

Below the steps I did to get this issue :

  1. Launch ZooKeeper
  2. Launch Kafka : .\bin\windows\kafka-server-start.bat .\config\server.properties

And at the second step the error happens :

ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) kafka.common.InconsistentClusterIdException: The Cluster ID Reu8ClK3TTywPiNLIQIm1w doesn't match stored clusterId Some(BaPSk1bCSsKFxQQ4717R6Q) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong. at kafka.server.KafkaServer.startup(KafkaServer.scala:220) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44) at kafka.Kafka$.main(Kafka.scala:84) at kafka.Kafka.main(Kafka.scala)

When I trigger .\bin\windows\kafka-server-start.bat .\config\server.properties zookeeper console returns :

INFO [SyncThread:0:FileTxnLog@216] - Creating new log file: log.1

How to fix this issue to get kafka running ?

Edit You can access to the proper issue on the right site (serverfault) here

Edit Here is the Answer

like image 725
TourEiffel Avatar asked Dec 25 '19 21:12

TourEiffel


People also ask

Does Kafka 3.1 need zookeeper?

In Kafka architecture, Zookeeper serves as a centralized controller for managing all the metadata information about Kafka producers, brokers, and consumers. However, you can install and run Kafka without Zookeeper.

Can we start Kafka server without starting zookeeper?

So to run kafka without zookeeper, it can be using with Kafka Raft metadata mode. We can say shortly, KRaft. It provides to us the kafka metadata information will be stored as a partition within kafka itself.

Is Zookeeper deprecated in Kafka?

Confluent views ZooKeeper's deprecation as an important move for the Kafka community, said Jun Rao, Kafka's co-creator and co-founder of Confluent. “It makes deployment/operation much simpler and improves the scalability by a factor of 10 because of more efficient handling of metadata.


2 Answers

I managed to Solve this issue with the following steps :

  1. Just Delete all the log/Data file created (or generated) into zookeeper and kafka.
  2. Run Zookeper
  3. Run Kafka

[Since this post is open again I post my answer there so you got all on the same post]

like image 188
TourEiffel Avatar answered Oct 02 '22 04:10

TourEiffel


** 1. The easiest solution is to remove kafka logs and start again.

** 2. But the root cause is Kafka saved failed cluster ID in meta.properties.**

Try to delete kafka-logs/meta.properties from your tmp folder, which is located in C:/tmp folder by default on windows, and /tmp/kafka-logs on Linux

if kafka is running in docker containers, the log path may be specified by volume config in the docker-compose - see docs.docker.com/compose/compose-file/compose-file-v2/#volumes -- Chris Halcrow

** 3. How to find Kafka log path:**

Open server server.properties file which is located in your kafka folder kafka_2.11-2.4.0\config\server.properties (considering your version of kafka, folder name could be kafka_<kafka_version>):

Then search for entry log.dirs to check where logs locate log.dirs=/tmp/kafka-logs

like image 40
Haili Sun Avatar answered Oct 02 '22 04:10

Haili Sun