Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

kafka get partition count for a topic

How can I get number of partitions for any kafka topic from the code. I have researched many links but none seem to work.

Mentioning a few:

http://grokbase.com/t/kafka/users/148132gdzk/find-topic-partition-count-through-simpleclient-api

http://grokbase.com/t/kafka/users/151cv3htga/get-replication-and-partition-count-of-a-topic

http://qnalist.com/questions/5809219/get-replication-and-partition-count-of-a-topic

which look like similar discussions.

Also there are similar links on SO which do not have a working solution to this.

like image 368
vish4071 Avatar asked Feb 16 '16 16:02

vish4071


People also ask

How many partitions can a Kafka topic have?

But here are a few general rules: maximum 4000 partitions per broker (in total; distributed over many topics) maximum 200,000 partitions per Kafka cluster (in total; distributed over many topics) resulting in a maximum of 50 brokers per Kafka cluster.

How many partitions are in a Kafka cluster?

Cluster guidelines A Kafka cluster should have a maximum of 200,000 partitions across all brokers when managed by Zookeeper. The reason is that if brokers go down, Zookeeper needs to perform a lot of leader elections. Confluent still recommends up to 4,000 partitions per broker in your cluster.

How do I check topic details in Kafka?

You can use the bin/kafka-topics.sh shell script along with the Zookeeper service URL as well as the –list option to display a list of all the topics in the Kafka cluster. You can also pass the Kafka cluster URL to list all topics.


2 Answers

Go to your kafka/bin directory.

Then run this:

./kafka-topics.sh --describe --zookeeper localhost:2181 --topic topic_name 

You should see what you need under PartitionCount.

Topic:topic_name        PartitionCount:5        ReplicationFactor:1     Configs:         Topic: topic_name       Partition: 0    Leader: 1001    Replicas: 1001  Isr: 1001         Topic: topic_name       Partition: 1    Leader: 1001    Replicas: 1001  Isr: 1001         Topic: topic_name       Partition: 2    Leader: 1001    Replicas: 1001  Isr: 1001         Topic: topic_name       Partition: 3    Leader: 1001    Replicas: 1001  Isr: 1001         Topic: topic_name       Partition: 4    Leader: 1001    Replicas: 1001  Isr: 1001 

When using a version where zookeeper is no longer a dependency of Kafka

kafka-topics --describe --bootstrap-server localhost:9092 --topic topic_name 
like image 187
peter.petrov Avatar answered Oct 14 '22 02:10

peter.petrov


In the 0.82 Producer API and 0.9 Consumer api you can use something like

Properties configProperties = new Properties(); configProperties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"localhost:9092"); configProperties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.ByteArraySerializer"); configProperties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringSerializer");  org.apache.kafka.clients.producer.Producer producer = new KafkaProducer(configProperties); producer.partitionsFor("test") 
like image 38
Sunil Patil Avatar answered Oct 14 '22 02:10

Sunil Patil