Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to Delete a topic in apache kafka [duplicate]

I need to delete a topic in kafka-0.8.2.2.3. I have used the below command for deleting the topic:

bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic DummyTopic 

The command executed successfully but when I run a command to list the topics, I could see that the topic is still there and it shows marked for deletion.

bin/kafka-topics.sh --list --zookeeper localhost:2181 DummyTopic - marked for deletion 

And when I create the topic DummyTopic it outputs the exception, The topic already exists, below is the stack trace:

Error while executing topic command Topic "DummyTopic" already exists. kafka.common.TopicExistsException: Topic "DummyTopic" already exists.     at kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:248)     at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:233)     at kafka.admin.TopicCommand$.createTopic(TopicCommand.scala:92)     at kafka.admin.TopicCommand$.main(TopicCommand.scala:54)     at kafka.admin.TopicCommand.main(TopicCommand.scala) 

Please let me know how can I delete this topic.

like image 667
Rishi Arora Avatar asked Nov 05 '15 06:11

Rishi Arora


People also ask

Does Kafka remove duplicate messages?

Specifically, if Apache Kafka invokes a message handler more than once for the same message, it detects and discards any duplicate messages produced by the handler. The message handler will still execute the database transaction repeatedly.

How does Kafka handle duplicate records?

2.1 Write idempotent message handler It's the easiest way to have a deal with duplicate messages. The message handler is idempotent if calling it multiple times with the same payload has no additional effect. For example, modify an already modified Order with the same payload should give the same result.

Can I delete __ Consumer_offsets topic?

__consumer_offsets is a kafka internal topic and it is not allowed to be deleted through delete topic command. It contains information about committed offsets for each topic:partition for each group of consumers (groupID). If you want to wipe it out entirely you have to delete the zookeeper dataDir location.


1 Answers

Deletion of a topic has been supported since 0.8.2.x version. You have to enable topic deletion (setting delete.topic.enable to true) on all brokers first.

Note: Ever since 1.0.x, the functionality being stable, delete.topic.enable is by default true.

Follow this step by step process for manual deletion of topics

  1. Stop Kafka server
  2. Delete the topic directory, on each broker (as defined in the logs.dirs and log.dir properties) with rm -rf command
  3. Connect to Zookeeper instance: zookeeper-shell.sh host:port
  4. From within the Zookeeper instance:
    1. List the topics using: ls /brokers/topics
    2. Remove the topic folder from ZooKeeper using: rmr /brokers/topics/yourtopic
    3. Exit the Zookeeper instance (Ctrl+C)
  5. Restart Kafka server
  6. Confirm if it was deleted or not by using this command kafka-topics.sh --list --zookeeper host:port
like image 112
Ravindra babu Avatar answered Sep 20 '22 06:09

Ravindra babu