Running into LeaderNotAvailableException when using Kafka 0.8.1 with Zookeeper 3.4.6

Question

I installed the stable version of kafka (0.8.1 with 2.9.2 Scala) as per their website and am running it with a 3 node zookeeper ensemble (3.4.6). I tried to create a test topic but keep seeing that there is no leader assigned to the partition of the topic:

[kafka_2.9.2-0.8.1]$ ./bin/kafka-topics.sh --zookeeper <zookeeper_ensemble> --describe --topic test-1 Topic:test-1    PartitionCount:1    ReplicationFactor:3 Configs:     Topic: test-1   Partition: 0    **Leader: none**    Replicas: 0,1,2 **Isr:**  

I tried to write to the topic anyway using the console producer but ran into the LeaderNotAvailableException exception:

[kafka_2.9.2-0.8.1]$ ./kafka-console-producer.sh --broker-list <broker_list> --topic test-1  hello world  [2014-04-22 11:58:48,297] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->  No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)  [2014-04-22 11:58:48,321] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->  No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)  [2014-04-22 11:58:48,322] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-1 (kafka.producer.async.DefaultEventHandler)  [2014-04-22 11:58:48,445] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->  No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)  [2014-04-22 11:58:48,467] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->  No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)  [2014-04-22 11:58:48,467] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-1 (kafka.producer.async.DefaultEventHandler)  [2014-04-22 11:58:48,590] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->  No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)  [2014-04-22 11:58:48,612] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->  No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)  [2014-04-22 11:58:48,612] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-1 (kafka.producer.async.DefaultEventHandler)  [2014-04-22 11:58:48,731] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->  No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)  [2014-04-22 11:58:48,753] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->  No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)  [2014-04-22 11:58:48,754] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test-1 (kafka.producer.async.DefaultEventHandler)  [2014-04-22 11:58:48,876] WARN Error while fetching metadata [{TopicMetadata for topic test-1 ->  No partition metadata for topic test-1 due to kafka.common.LeaderNotAvailableException}] for topic [test-1]: class kafka.common.LeaderNotAvailableException  (kafka.producer.BrokerPartitionInfo)  [2014-04-22 11:58:48,877] ERROR Failed to send requests for topics test-1 with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)  [2014-04-22 11:58:48,878] ERROR Error in handling batch of 1 events (kafka.producer.async.ProducerSendThread) kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.     at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)     at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)     at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)     at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)     at scala.collection.immutable.Stream.foreach(Stream.scala:547)     at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)     at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44) 

I should also state that this was working initially for a few days and then suddenly any topic that was created had this missing leader problem.

Answer 1

Kafka uses an external coordination framework (by default Zookeeper) to maintain configuration. It seems the configuration is now out-of-sync with the Kafka log data. In this case, I'd remove affected topic data and related Zookeeper data.

For Test Environment:

1. Stop Kafka-server and Zookeeper-server
2. Remove the data directories of both services, by default they are /tmp/kafka-log and /tmp/zookeeper.
3. Start Kafka-server and Zookeeper-server again
4. Create a new topic

Now you are able to work with the topic again.

For Production Environment:

As the Kafka topics are stored in different directories, you should remove particular directories. You should also remove /brokers/{broker_id}/topics/{broken_topic} from Zookeeper by using a Zookeeper client.

Please read Kafka documentation carefully to make sure the configuration structure, before you do anything stupid. Kafka is rolling out a delete topic feature (KAFKA-330), so that the problem can be solved more easily.