Can a Kafka producer create topics and partitions?

Question

currently I am evaluating different Messaging Systems. There is a question related to Apache Kafka which I could not answer myself.

Is it possible for a Kafka producer to create topics and partitions (on existing topics aswell) dynamically? If yes, is there any disadvantage that comes with it?

Thanks in Advance

Answer 1

Updated:

The kafka broker has a property: auto.create.topics.enable

If you set that to true if the producer publishes a message to the topic with the new topic name it will automatically create a topic for you.

The Confluent Team recommends not doing this because the explosion of topics, depending on your environment can become unwieldy, and the topic creation will always have the same defaults when created. It's important to have a replication-factor of at least 3 to ensure durability of your topics in the event of disk failure.

Answer 2

When you are starting your kafka broker you can define a bunch of properties in conf/server.properties file. One of the property is auto.create.topics.enable if you set this to true (by default) kafka will automatically create a topic when you send a message to a non existing topic. The partition number will be defined by the default settings in this same file.

Disadvantages : as far as I know, topics created this way will always have the same default settings (partitions, replicas ...).