How can I get number of partitions for any kafka topic from the code. I have researched many links but none seem to work.
Mentioning a few:
http://grokbase.com/t/kafka/users/148132gdzk/find-topic-partition-count-through-simpleclient-api
http://grokbase.com/t/kafka/users/151cv3htga/get-replication-and-partition-count-of-a-topic
http://qnalist.com/questions/5809219/get-replication-and-partition-count-of-a-topic
which look like similar discussions.
Also there are similar links on SO which do not have a working solution to this.
But here are a few general rules: maximum 4000 partitions per broker (in total; distributed over many topics) maximum 200,000 partitions per Kafka cluster (in total; distributed over many topics) resulting in a maximum of 50 brokers per Kafka cluster.
Cluster guidelines A Kafka cluster should have a maximum of 200,000 partitions across all brokers when managed by Zookeeper. The reason is that if brokers go down, Zookeeper needs to perform a lot of leader elections. Confluent still recommends up to 4,000 partitions per broker in your cluster.
You can use the bin/kafka-topics.sh shell script along with the Zookeeper service URL as well as the –list option to display a list of all the topics in the Kafka cluster. You can also pass the Kafka cluster URL to list all topics.
Go to your kafka/bin
directory.
Then run this:
./kafka-topics.sh --describe --zookeeper localhost:2181 --topic topic_name
You should see what you need under PartitionCount
.
Topic:topic_name PartitionCount:5 ReplicationFactor:1 Configs: Topic: topic_name Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: topic_name Partition: 1 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: topic_name Partition: 2 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: topic_name Partition: 3 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: topic_name Partition: 4 Leader: 1001 Replicas: 1001 Isr: 1001
When using a version where zookeeper is no longer a dependency of Kafka
kafka-topics --describe --bootstrap-server localhost:9092 --topic topic_name
In the 0.82 Producer API and 0.9 Consumer api you can use something like
Properties configProperties = new Properties(); configProperties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"localhost:9092"); configProperties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.ByteArraySerializer"); configProperties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,"org.apache.kafka.common.serialization.StringSerializer"); org.apache.kafka.clients.producer.Producer producer = new KafkaProducer(configProperties); producer.partitionsFor("test")
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With