Error creating Kafka topic :- replication factor larger than available brokers

Question

I am trying to create a kafka topic via AdminCommand using below code Source

 ZkClient zkClient = new ZkClient(kafkaHost, 10000, 10000, ZKStringSerializer$.MODULE$);
    AdminUtils.createTopic(zkClient, "pa_reliancepoc_telecom_usageevent", 10, 2, new Properties());

But getting the below exception

Exception in thread "main" kafka.admin.AdminOperationException: replication factor: 1 larger than available brokers: 0
at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:70)
at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:155)

However, I am able to create the topic using shell command .

Answer 1

In your code,

 AdminUtils.createTopic(zkClient, "pa_reliancepoc_telecom_usageevent", 10, 2, new Properties());

The fourth argument is the replication factor. So you are trying to create a topic with a name of pa_reliancepoc_telecom_usageevent with partition count of 10 and replication of 2. So two kafka brokers should be available while creating the topic. If less than two is available then you will get the following exception.

Exception in thread "main" kafka.admin.AdminOperationException: replication factor: 1 larger than available brokers: 0
at kafka.admin.AdminUtils$.assignReplicasToBrokers(AdminUtils.scala:70)
at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:155)

Make sure that you are running kafka cluster with two broker nodes and the two nodes should be alive while creating the topic.

To run kafka in cluster refer Step 6 in this link

Answer 2

configure your local machine to have multiple brokers up and running in case you decide to keep your replication_factor > 1.

You can do that by simply having multiple copies of the server.properties file. e.g
server-1.properties & server-2.properties

You need to then specify different broker.id & port in each of those files to make them unique ..

   config/server-1.properties:
     broker.id=1
     port=9093
     log.dir=/tmp/kafka-logs-1

   config/server-2.properties:
     broker.id=2
     port=9094
     log.dir=/tmp/kafka-logs-2

And then start multiple instances with following commands

> bin/kafka-server-start.sh config/server-1.properties &
> bin/kafka-server-start.sh config/server-2.properties &

For more details check Step 6: Setting up a multi-broker cluster