How to decrease number partitions Kafka topic?

Question

I created a topic with 4 partitions on Kafka. (set default number.partition=4). Now I want to change number partition of this topic to 3. I've tried running

./bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic my-topic --partitions 3 

but there is no change. It still has 4 partitions. Anyone know about this?

Answer 1

Apache Kafka doesn't support decreasing the partition number. You should see the topic as a whole and the partitions are a way for scaling out improving performance. So all data sent to topic flow to all partitions and removing one of them means data loss.

Answer 2

You can't just delete a partition because that would lead to data loss and also the remaining data's keys would not be distributed correctly so new messages would not get directed to the same partitions as old existing messages with the same key.

For the above reasons Kafka does not support decreasing partition counts on an existing topic.

What you can do is to create a new topic with 3 partitions and then write an small program (or use an existing replication tool) to copy the data from the old 4 partition topic to the new 3 partition topic. That way you will be running everything through the same partitioner and all your keyed messages will end up in the right partition. Once you are satisfied the data is all copied then delete the original 4 partition topic.

If you must retain the same topic name as the original topic then just create a new topic with the original name, copy the data back from the repartitioned topic, and delete that temporary repartitioning topic.