Menu
I use Ubuntu server 16.04 to try using Kafka. For the command to start a producer and a consumer console I use the following.
producer console :
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic hello-topic
consumer console :
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic hello-topic
but the command above only subscribes to one topic. How can I subscribe to multiple topics?
717
A. Yes, Kafka's design allows consumers from one consumer group to consume messages from multiple topics.
Multi-Topic Consumers We may have a consumer group that listens to multiple topics. If they have the same key-partitioning scheme and number of partitions across two topics, we can join data across the two topics.
There is no need for multiple threads, you can have one consumer, consuming from multiple topics. Offsets are maintained by zookeeper, as kafka-server itself is stateless. Whenever a consumer consumes a message,its offset is commited with zookeeper to keep a future track to process each message only once.
So the rule in Kafka is only one consumer in a consumer group can be assigned to consume messages from a partition in a topic and hence multiple Kafka consumers from a consumer group can not read the same message from a partition.
First you should connect with the option bootstrap-server to the Kafka server itself not the zookeeper server.
For multiple topics you use the whitelist option. This will get interpreted as a regular expression and has to get quoted, see Kafka documentation. So a correct command would be:
kafka-console-consumer.sh --bootstrap-server localhost:9092 --whitelist 'hello-topic|world-topic|another-topic'
Other expressions are also possible, like
kafka-console-consumer.sh --bootstrap-server localhost:9092 --whitelist '.*'
BE AWARE
For convenience we allow the use of ',' instead of '|' to specify a list of topics.
Does not work with Kafka 2.0, perhaps only when mirroring, which I did not try yet.
165
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
