Menu
I'm using kafka-python and I'm wondering if there is a way for showing all the topics.
Something like this:
./bin/kafka-topics.sh --list --zookeeper localhost:2181
638
You can use the bin/kafka-topics.sh shell script along with the Zookeeper service URL as well as the –list option to display a list of all the topics in the Kafka cluster. You can also pass the Kafka cluster URL to list all topics.
Create a file named consumer1.py with the following python script. KafkaConsumer module is imported from the Kafka library to read data from Kafka. sys module is used here to terminate the script. The same hostname and port number of the producer are used in the script of the consumer to read data from Kafka.
kafka-python is designed to function much like the official java client, with a sprinkling of pythonic interfaces (e.g., consumer iterators). kafka-python is best used with newer brokers (0.9+), but is backwards-compatible with older versions (to 0.8. 0). Some features will only be enabled on newer brokers.
You can count the number of messages in a Kafka topic simply by consuming the entire topic and counting how many messages are read. To do this from the commandline you can use the kcat tool which can act as a consumer (and producer) and is built around the Unix philosophy of pipelines.
You have to install the necessary python library to read data from Kafka. Python3 is used in this tutorial to write the script of consumer and producer.
Listing Kafka Topics 1 Overview. In this quick tutorial, we're going to see how we can list all topics in an Apache Kafka cluster. ... 2 Setting Up Kafka. First, we should make sure to download the right Kafka version from the Apache site. ... 3 Listing Topics. ... 4 Topic Details. ... 5 Conclusion. ...
Kafka is an open-source distributed messaging system to send the message in partitioned and different topics. Real-time data streaming can be implemented by using Kafka to receive data between the applications.
The Apache Kafka binaries are also a set of useful command-line tools that allow us to interact with Kafka and Zookeeper via the command line. If you don't have them, you can download them from the official Apache Kafka Downloads repository.
import kafka
consumer = kafka.KafkaConsumer(group_id='test', bootstrap_servers=['server'])
consumer.topics()
Try with the method KafkaConsumer.topics().
25
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
