Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to set auto.create.topics.enable as default config on AWS MSK cluster

Our AWS MSK cluster is not automatically creating topics when a producer first publishes a message to the topic. This is surprising because the default behaviour according to https://kafka.apache.org/documentation/#brokerconfigs should have auto.create.topics.enable = true.

I have attempted to set this configuration on my brokers but I am unable because the config is read-only for dynamic updates.

$ kafka-configs --bootstrap-server $KAFKA_BROKER --entity-type brokers --entity-default --alter --add-config auto.create.topics.enable=true

Error while executing config command with args '--bootstrap-server $KAFKA_BROKER --entity-type brokers --entity-default --alter --add-config auto.create.topics.enable=true'
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Invalid config value for resource ConfigResource(type=BROKER, name=''): Cannot update these configs dynamically: Set(auto.create.topics.enable)

note: I replaced our broker's IP with $KAFKA_BROKER in the output

How can I configure my AWS MSK Kafka cluster to enable auto creation of topics?

like image 309
rpassza Avatar asked Apr 26 '19 09:04

rpassza


People also ask

How do you create a topic in MSK Kafka?

To create a topic on the client machineIn the navigation pane, choose Instances. Then select the check box beside the name of the client machine that you created in Step 2: Create a client machine. Choose Actions, and then choose Connect. Follow the instructions in the console to connect to your client machine.

How long does it take to create an MSK cluster?

Description. New AWS MSK Clusters can routinely take close to 60 minutes or more to be created.


2 Answers

Update

You can now customise MSK configuration:

auto.create.topics.enable: Enables topic autocreation on the server.

Therefore, auto.create.topics.enable=true (defaults to false) should now do the trick as usual.


Although auto.create.topics.enable normally defaults to true, this is not the case for AWS MSK. According to these slides presented in Berlin's AWS Kafka Summit this February, the following are the default configuration parameters for Amazon MSK:

offsets.topic.replication.factor 3
transaction.state.log.replication.factor 3
transaction.state.log.min.isr 2
auto.create.topics.enable False
default.replication.factor 3
min.insync.replicas 2
unclean.leader.election.enable True
auto.leader.rebalance.enable True
authorizer.class.name kafka.security.auth.SimpleAclAuthorizer
group.initial.rebalance.delay.ms 3000
log.retention.hours 168 

Note that even Kafka docs mention that the suggested production configuration should be set to false.

like image 137
Giorgos Myrianthous Avatar answered Oct 23 '22 13:10

Giorgos Myrianthous


Moving forward you can now set this parameter to true by customizing the configuration, however it will stay false by default since that is the recommended production configuration.

like image 39
Eno Thereska Avatar answered Oct 23 '22 13:10

Eno Thereska