I installed Kafka 2.4.0 and am seeing odd behavior when I run the kafka-topics.sh command.
I can start zookeeper and kafka with no issues, and when I run this
kafka-topics.sh --bootstrap-server 127.0.0.1:2181 --version
I get this:
2.4.0 (Commit:77a89fcf8d7fa018)
So far so good. But if I try to list all the topics:
kafka-topics.sh --bootstrap-server 127.0.0.1:2181 --list
the command just hangs and I see the zookeeper console spews this message repeatedly:
WARN Exception causing close of session 0x0: Unreasonable length = 308375649
At first I thought this may be a clash with Confluent 5.3.2, which I also had installed, but I removed that. I uninstalled Kafka, and re-installed. No joy.
cleanup. policy property from topic config which by default is delete , says that "The delete policy will discard old segments when their retention time or size limit has been reached." So, if you send record with producer api and topic got full, it will discard old segments.
The rule of thumb is that the number of Kafka topics can be in the thousands. Jun Rao (Kafka committer; now at Confluent but he was formerly in LinkedIn's Kafka team) wrote: At LinkedIn, our largest cluster has more than 2K topics. 5K topics should be fine.
Kafka uses ZooKeeper to store its metadata about partitions and brokers, and to elect a broker to be the Kafka Controller. Currently, removing this dependency on ZooKeeper is work in progress (through the KIP-500) .
kafka-topics.sh --bootstrap-server 127.0.0.1:2181 --list
The problem is that you are using zookeeper address as a bootstrap-server address. It shoud be like this:
kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --list
Alternatively you can also use zookeeper address too with --zookeeper parameter like this:
kafka-topics.sh --zookeeper 127.0.0.1:2181 --list
P.S: Prior to Kafka version 2.2 zookeeper is the only valid option in topics.sh, but after that both bootstrap-server and zookeeper are valid options. You can check this for more information.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With