How can I define two or more Kafka consumers using spring boot configurations?

Question

I'm new to kafka. How can I define two(or more) Kafka consumers using configuration .properties or .yml? I'm interested in using the spring.kafka.* application's properties and also I would like to specify two different properties for two consumers using this configuration files. For example, consumerA will have spring.kafka.bootstrap-servers=localhost:9090 and consumerB spring.kafka.bootstrap-servers=localhost:9091. I have seen examples online using multiple @KafkaLister with a single application.yml where all the common properties of the @KafkaLister (consumer) beans are defined in application.yml. This is ok concept but in case there are more consumers and they have completely different configurations then all of the configs needs to be put in the @KafkaListener annotation and this will make the class long and hard to read and the .yml obsolete. Instead, I would like something like this:

spring:
  kafka:
    consumer1:
      auto-offset-reset: earliest
      spring.kafka.bootstrap-servers=localhost:9091
  kafka:
    consumer2:
      spring.kafka.bootstrap-servers=localhost:9092
      auto-offset-reset: latest
  kafka:
    consumer3:
      spring.kafka.bootstrap-servers=localhost:9093
      auto-offset-reset: latest

And also how would I then connect this configuration to the appropriate beans? I could surely define the consumer beans and then use my custom configuration files to create as many different consumers as I would like but it seems to me that I would be reinventing the wheel.

Thanks

Answer 1

No; you can't declare consumers like that, you have to use @KafkaListener annotations on methods or create listener containers/listeners yourself.

You can override the bootstrap servers in the listener itself...

@KafkaListener(... properties="bootstrap.servers:${some.property}")
void listen(...) {

}

You can programmatically make multiple consumers from user properties, using the technique described here...

Can i add topics to my @kafkalistener at runtime