Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to pass multiple bootstrap servers for listener using spring-kafka

I have a listener that need to read from multiple kafka servers with the same topic which are all configured under one zookeeper. How do I read from those multiple servers. Can you please help with this.

Instead of Kafka servers, can I point to zookeeper instead?

like image 628
daemon54 Avatar asked Sep 19 '17 13:09

daemon54


2 Answers

The @KafkaListener requires KafkaListenerContainerFactory @Bean, which, in turn, is based on the ConsumerFactory. And the DefaultKafkaConsumerFactory accepts a Map<String, Object> of consumer configs:

@Configuration
@EnableKafka
public class KafkaConfig {

    @Bean
    KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<Integer, String>>
                        kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<Integer, String> factory =
                                new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        return factory;
    }

    @Bean
    public ConsumerFactory<Integer, String> consumerFactory() {
        return new DefaultKafkaConsumerFactory<>(consumerConfigs());
    }

    @Bean
    public Map<String, Object> consumerConfigs() {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, ...);
        ...
        return props;
    }
}

https://docs.spring.io/spring-kafka/docs/1.2.2.RELEASE/reference/html/_reference.html#__kafkalistener_annotation

Where that ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG is exactly standard Apache Kafka bootstrap.servers property:

A list of host/port pairs to use for establishing the initial connection to the Kafka cluster. The client will make use of all servers irrespective of which servers are specified here for bootstrapping—this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form host1:port1,host2:port2,.... Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down).

No, you can't point Zookeeper address. That isn't supported by Kafka any more.

like image 54
Artem Bilan Avatar answered Oct 21 '22 04:10

Artem Bilan


The map has key: String and Value: Object.

The Object of key:ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG should be series of host-port pair seperated by ',' like: host1:port1,host2:port2,host3:port3 ......

e.g for listening three servers: localhost:9092,192.168.22.12:9088,localhost:7898

like image 32
Karthikumar Avatar answered Oct 21 '22 03:10

Karthikumar