Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Kafka create topic with default number of partitions

Tags:

apache-kafka

I am trying to create a new Kafka topic from the command line

$ kafka-topics --create --zookeeper localhost:2181 --topic def-test 

I get the error

Missing required argument "[partitions]"

From the docs, I see that setting num.partitions, should have done the trick. I have the following in my server.properties

# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=2

But it is not taking effect. Also, I wonder how kafka-topics command which connects only to zookeeper and does not take any arguments to server.properties is going to be able to pick the correct value. How can I create topics without having to specify the number of partitions (by falling back to a default value specified elsewhere)?

like image 323
senseiwu Avatar asked Feb 26 '18 10:02

senseiwu


People also ask

How to specify the number of partitions in a Kafka topic?

To specify the number of partitions, you should create the topic from CLI instead of expecting Kafka to create it when you first publish messages. To alter the number of partitions in an already existed topic:

What is offset in Kafka partitioning?

The records in the partitions are each assigned a sequential identifier called the offset, which is unique for each record within the partition. The offset is an incremental and immutable number, maintained by Kafka. When a record is written to a partition, it is appended to the end of the log, assigning the next sequential offset.

Where does Kafka store the data?

In Kafka, the data is store in the Kafka topics. The Kafka topic will further be divided into multiple partitions. The actual messages or the data will store in the Kafka partition. It is directly proportional to the parallelism.

What happens when you select * from S1 in Kafka?

The result of SELECT * FROM S1 causes every record from Kafka topic topic1 (with 1 partition and 1 replica) to be produced to Kafka topic topic2 (with 2 partitions and 2 replicas). Describe the properties of the new topic, topic2, underlying the ksqlDB stream you just created.


1 Answers

I was trying the quickstart guide of Kafka and was facing this issue. As the guide suggests, I ran the following command,

$ bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092

and got the following error:

Missing required argument "[partitions]"

As the error clearly states, we need to add more arguments to the command we use. For this you need to add --partitions 1. After this is added you will get the following error.

Missing required argument "[replication-factor]"

Do the same to this as well. Add the flag --replication-factor 1. So finally my command would look like

bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

I hope this helps someone who is stuck with the quickstart guide. More on what these flags mean is given below.


enter image description here enter image description here

like image 145
Keet Sugathadasa Avatar answered Dec 10 '22 19:12

Keet Sugathadasa