How to check the existence of Kafka topic in Nodejs

Question

I am currently working with Nodejs and Kafka whereby a Nodejs server is set up to receive events and the data corresponding to the events are sent to Kafka. In Kafka, producer will create a topic accordingly and dynamically, if topic does not exist already. To do that, I want to check the existence of the topic, if it exists or not before creation.

I am currently using kafka-node module for kafka-node integrated functionalities. However, I could not find out any functionality which either tells about the existence of the topic or returns the list of all the topics currently exist in kafka.

On searching on Internet, I found kafka-rest proxy which does help to know about this by fetching the current topics but I did not understand how to put that to use much.

Is there any other API via which I can achieve the stated functionality above?

Answer 1

You can do both at the same time, actually. Just use the undocumented Client.loadMetadataForTopics() function. Like this:

var kafka = require('kafka-node');
var client = new kafka.Client("localhost:2181");

client.loadMetadataForTopics(["NonExistentTopic"], (err, resp) => {
  console.log(JSON.stringify(resp))
});

// [{"0":{"nodeId":0,"host":"host-001","port":9092}},{"error":["LeaderNotAvailable"],"metadata":{}}]

See the LeaderNotAvailable error? That means the topic doesn't exist. But -- assuming auto.topic.create.enable is set to true, then the call to loadMetadataForTopics will also create the topic. So you get both in one call -- if you get the error, then the topic didn't exist and was created, if you don't get the error you get the actual topic metadata.