Leader brokers without a matching listener error in kafka

Question

What is the meaning of this kafka error ?

[2018-08-22 11:40:49,429] WARN [Consumer clientId=consumer-1, groupId=console-consumer-62114] 1 partitions have leader brokers without a matching listener, including [topicname-0] (org.apache.kafka.clients.NetworkClient)

I'm getting it when running:

./kafka-console-consumer.sh --topic topicname --bootstrap-server localhost:9094

And I'm getting some errors inside a golang program when trying to read this topic:

2018/08/22 11:44:12 ReadOffsetWithRetryOnError conn error: < dial tcp :0: connect: connection refused > kafka0:9094 topic: 0

The code snippet:

conn, err := kafka.DialLeader(context.Background(), "tcp", ip, getTopic(topic), 0)
                if err != nil {
                    log.Println("ReadOffsetWithRetryOnError conn error: <", err, "> ", ip, " topic:", topic)
                }

This is quite weird because, when reading on different topic it is working fine at the same time.

More error logs:

/kafka-topics.sh --describe --zookeeper localhost:2181 --topic topicname Topic:indexBlock PartitionCount:1
ReplicationFactor:1 Configs: Topic: topicname Partition: 0 Leader: -1 Replicas: 1002 Isr: 1002

Answer 1

[2018-08-22 11:40:49,429] WARN [Consumer clientId=consumer-1, groupId=console-consumer-62114] 1 partitions have leader brokers without a matching listener, including [topicname-0] (org.apache.kafka.clients.NetworkClient)

This error also happens if you try to run multiple consumers and the kafka topic contains only one partition. Generally one consumer should mapped with one partition. If you are using two consumers then you should have 2 partition in the kafka topic.

Answer 2

In my case, i was getting this error when i was testing Kafka fail-over. I brought down 1 Kafka, and expected the message to be written to the other Kafka.

The issue was that topic replication-factor was set to 1, when i needed to set it to 2. (2 Kafka instances)

Bonus:
Check out the directories where the topics are created(in my case: kafka-logs-xx) for both Kafka, and you will understand why :-)