Managing Kafka Topic with spring

Question

We are planning to use Kafka for queueing in our application. I have some bit of experience in RabbitMQ and Spring.

With RabbitMQ and Spring, we used to manage queue creation while starting up the spring service.

With Kafka, I'm not sure what could be the best way to create the topics? Is there a way to manage the topics with Spring.

Or, should we write a separate script which helps in creating topics? Maintaining a separate script for creating topics seems a bit weird for me.

Any suggestions will be appreciated.

Answer 1

To automatically create a Kafka topic in Spring Boot, only this is required:

@Bean
public NewTopic topic1() {
    return new NewTopic("foo", 10, (short) 2);

    //foo: topic name
    //10: number of partitions
    //2: replication factor
}

The Kafka Admin is being automatically created and configured by Spring Boot.

Version 2.3 of Spring Kafka introduced a TopicBuilder class, to make building topics fluent and more intuitive:

@Bean
public NewTopic topic(){
    return TopicBuilder.name("foo")
        .partitions(10)
        .replicas(2)
        .build();
}

Answer 2

In spring it is possible to create topics during the start of the application using beans:

@Bean
public KafkaAdmin admin() {
    Map<String, Object> configs = new HashMap<>();
    configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG,
            StringUtils.arrayToCommaDelimitedString(kafkaEmbedded().getBrokerAddresses()));
    return new KafkaAdmin(configs);
}

@Bean
public NewTopic topic1() {
    return new NewTopic("foo", 10, (short) 2);
}

Alternatively you can write your own create topics by autowiring the AdminClient, so for instance reading the list from an input file or specify advanced properties such as partition numbers:

@Autowired
private KafkaAdmin admin;
//...your implementation

Also note that since Kafka 1.1.0 auto.create.topics.enable is enabled by default (see Broker configs).

For more information refer to the spring-kafka docs