Re-assignment of Partition gets infinitely stuck in "Still in progress" state

Question

We have 6 node kafka cluster. Out of the 6 machines, 3 of them have both Kafka + zookeeper, and the remaining 3 of them have just kafka. Recently, we added one more kafka node. While re-assigning the partition to all the nodes (including the newer one), we executed the below command:

/opt/kafka/bin/kafka-reassign-partitions.sh --reassignment-json-file new_assignment_details.json --execute --zookeeper localhost:2181

However, when we verified the status by using the below command,

/opt/kafka/bin/kafka-reassign-partitions.sh --reassignment-json-file new_assignment_details.json --verify --zookeeper localhost:2181

We get the below output. Some of the partitions are re-assignment is still in progress.

/opt/kafka/bin/kafka-reassign-partitions.sh --reassignment-json-file new_assignment_details.json --verify --zookeeper localhost:2181 | grep 'progress'
Reassignment of partition [topic-name,854] is still in progress
Reassignment of partition [topic-name,674] is still in progress
Reassignment of partition [topic-name,944] is still in progress
Reassignment of partition [topic-name,404] is still in progress
Reassignment of partition [topic-name,314] is still in progress
Reassignment of partition [topic-name,853] is still in progress
Reassignment of partition [prom-metrics,403] is still in progress
Reassignment of partition [prom-metrics,134] is still in progress

There is no way to either:

1. Cancel the on-going partition re-assignment.
2. Rollback is also not possible. ( When we try doing that it says that "There is an existing assignment running."

Kafka Version: 0.10.1.1

Platform: Amazon Linux 1 x86_64 GNU/Linux

Can anyone please help me out in this?

Answer 1

You can cancel the on-going partition re-assignment by deleting the zk node: delete /admin/reassign_partitions. Then you can rollback or reassign with another json file.

Answer 2

In my case, for canceling current partition re-assignment and being able to perform another re-assignment you should:

1. Delete /admin/reassign_partitions zk node
2. Delete /controller zk node to re-elect the active controller

UPDATE
consider using --additional argument of kafka-reassign-partitions.sh!