How to efficiently create Kafka topics with testcontainers?

Question

I am using testcontainers.org with KafkaContainer.

Currently, I use kafka-topics to create a topic after starting the container:

kafkaContainer.execInContainer("/bin/sh", "-c", "/usr/bin/kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic topicName");

As this takes around 3-5 seconds per topic, I am wondering, if there is a more efficient way to create multiple topics. Or is there a simple switch to autocreate topics on demand?

Answer 1

For reference, using the AdminClient was the most efficient.

Here is an example:

  private static void createTopics(String... topics) {
    var newTopics =
        Arrays.stream(topics)
            .map(topic -> new NewTopic(topic, 1, (short) 1))
            .collect(Collectors.toList());
    try (var admin = AdminClient.create(Map.of(BOOTSTRAP_SERVERS_CONFIG, getKafkaBrokers()))) {
      admin.createTopics(newTopics);
    }
  }

  private static String getKafkaBrokers() {
    Integer mappedPort = kafkaContainer.getFirstMappedPort();
    return String.format("%s:%d", "localhost", mappedPort);
  }

Answer 2

1. Use wurstmeister/kafka container with KAFKA_CREATE_TOPICS environment variable

2. You could use a higher level Kafka client like Spring-Kafka or Dropwizard-Kafka which offer topic creation.

3. Otherwise, use AdminClient directly

It's recommended not to enable auto topic creation on the brokers because it then has a default partition count and replication factor