Kafka - Troubleshooting .NotEnoughReplicasException

Question

I started seeing the following error

[2020-06-12 20:09:01,324] ERROR [ReplicaManager broker=3] Error processing append operation on partition __consumer_offsets-10 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR Set(3) is insufficient to satisfy the min.isr requirement of 2 for partition __consumer_offsets-10    

My setup is having three brokers and all brokers are up. Couple of things i did before this error was about pop up

I configured min.isr to be 2 in all the brokers. I created a topic with replication factor 3 and starting producing the message from a producer with ack = 1 with two brokers down. I brought up all the brokers and started consumer.

1. How to go about troubleshooting this error
2. Consumer is also NOT be able to see this message ( not sure why, the message is supposed to be treated as "committed" as one broker was up when the producer was running)

Couple of facts

It is interesting to see rebalancing didnt happen yet WRT preferred leader starategy

$ kafka-topics --zookeeper 127.0.0.1:2181 --topic stock-prices --describe                             
  Topic: stock-prices     PartitionCount: 3       ReplicationFactor: 3    Configs: min.insync.replicas=2
  Topic: stock-prices     Partition: 0    Leader: 1       Replicas: 1,3,2 Isr: 1,2,3            
  Topic: stock-prices     Partition: 1    Leader: 1       Replicas: 2,1,3 Isr: 1,2,3            
  Topic: stock-prices     Partition: 2    Leader: 1       Replicas: 3,2,1 Isr: 1,2,3

Answer 1

Here's your problem:

You have set min.insync.replicas=2, which means you need at least two broker up and running to publish a message to a topic. If you let down 2 brokers, then you have only one left. Which means your insync.replica requirement is not fulfilled.

This has nothing to do with the Consumers, since this is about the brokers. When you set acks=1 that means your producer gets the acknowledgement when the message is published to one broker. (It will not acknowledge all the replicas are created).

So the problem is, you have your Producer, which gets acknowledged that the message is received, when a single broker (The leader) gets the message. But the leader cannot put replicas since there aren't any brokers up to sync.

One way to get this done is to set the acks=all, so your producer won't get acknowledged until all the replicas are done. It'll retry until the at least 2 in sync replicas are online.