Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

"unreasonable length" when running kafka-topics command

Tags:

apache-kafka

I installed Kafka 2.4.0 and am seeing odd behavior when I run the kafka-topics.sh command.

I can start zookeeper and kafka with no issues, and when I run this

kafka-topics.sh --bootstrap-server 127.0.0.1:2181 --version

I get this: 2.4.0 (Commit:77a89fcf8d7fa018)

So far so good. But if I try to list all the topics:

kafka-topics.sh --bootstrap-server 127.0.0.1:2181 --list

the command just hangs and I see the zookeeper console spews this message repeatedly:

WARN Exception causing close of session 0x0: Unreasonable length = 308375649

At first I thought this may be a clash with Confluent 5.3.2, which I also had installed, but I removed that. I uninstalled Kafka, and re-installed. No joy.

like image 579
M. Ferris Avatar asked Feb 03 '20 01:02

M. Ferris


People also ask

What happens if Kafka topic is full?

cleanup. policy property from topic config which by default is delete , says that "The delete policy will discard old segments when their retention time or size limit has been reached." So, if you send record with producer api and topic got full, it will discard old segments.

How many topics Kafka can handle?

The rule of thumb is that the number of Kafka topics can be in the thousands. Jun Rao (Kafka committer; now at Confluent but he was formerly in LinkedIn's Kafka team) wrote: At LinkedIn, our largest cluster has more than 2K topics. 5K topics should be fine.

How is ZooKeeper used in Kafka?

Kafka uses ZooKeeper to store its metadata about partitions and brokers, and to elect a broker to be the Kafka Controller. Currently, removing this dependency on ZooKeeper is work in progress (through the KIP-500) .


1 Answers

kafka-topics.sh --bootstrap-server 127.0.0.1:2181 --list

The problem is that you are using zookeeper address as a bootstrap-server address. It shoud be like this:

kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --list

Alternatively you can also use zookeeper address too with --zookeeper parameter like this:

kafka-topics.sh --zookeeper 127.0.0.1:2181 --list

P.S: Prior to Kafka version 2.2 zookeeper is the only valid option in topics.sh, but after that both bootstrap-server and zookeeper are valid options. You can check this for more information.

like image 54
H.Ç.T Avatar answered Nov 11 '22 11:11

H.Ç.T