Menu
What is optimal no. of partition for a topic that has 5 brokers and replication factor=3? Total there are only 6 topics in this cluster.
Machine configuration that each broker is running as follows.
Memory=16gb
Porcessor= Octacore, Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
Any challenges other than open file for 1000 partitions per topic??
295
For most implementations you want to follow the rule of thumb of 10 partitions per topic, and 10,000 partitions per Kafka cluster. Going beyond that amount can require additional monitoring and optimization. (You can learn more about Kafka monitoring here.)
Following are some general guidelines: A Kafka cluster should have a maximum of 200,000 partitions across all brokers when managed by Zookeeper. The reason is that if brokers go down, Zookeeper needs to perform a lot of leader elections. Confluent still recommends up to 4,000 partitions per broker in your cluster.
In Kafka, replication means that data is written down not just to one broker, but to many. The replication factor is a topic setting and is specified at topic creation time. A replication factor of 3 is a commonly used replication factor as it provides the right balance between broker loss and replication overhead.
Starting since Kafka 1.1.0 it's recommended to have up to 4'000 partitions per broker and up to 200'000 partitions per cluster. Read more here.
Pls, note that even if you can create a large number of partitions it doesn't mean that your hardware will be able to support it. Make sure you scale your cluster hardware(CPU, RAM, HDD etc) accordingly to your load.
193
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
