Kafka - why new topic partition leader is not elected?

Question

I have a kafka cluster of 3 nodes. When Node #3 dies, my _schemas topic stops functioning properly and I see this:

kafka-topics --zookeeper localhost:2181 --topic _schemas --describe
Topic:_schemas  PartitionCount:1        ReplicationFactor:2     Configs:cleanup.policy=compact
    Topic: _schemas Partition: 0    Leader: -1      Replicas: 3,2   Isr: 2

So it seems that node #3 is dead and that is what Leader: -1 refers to. But why doesn't Kafka just continue working as usual, assigning Node #2 as the new leader and replicating the data to #1 so that we have 2 in sync replicas?

The error I see in the kafka logs:

kafka.common.NotAssignedReplicaException:
Leader 3 failed to record follower 2's position -1 since the replica is not 
recognized to be one of the assigned replicas 3 for partition <loop over many partitions>

Answer 1

I solved this problem by restarting the controller broker. every kafka cluster has a broker been elected to controller so it will coordinate leader election. our case is that controller stuck. in order to find which broker is controller, you can just go to your zkCli.sh to access into your zookeeper which your kafka cluster uses, and then do get /controller, you will see brokerId there. I fixed this easily by restarting the controller broker, good luck.