Menu
Is it possible to read from multiple partitions using Kafka Simple Consumer? Simple Consumer uses the partition in the following:
PartitionMetadata metadata = findLeader(brokers, port, topic, partition);
SimpleConsumer consumer = new SimpleConsumer(leadBroker, port, 100000, 64 * 1024, clientName);
leadBroker = findNewLeader(leadBroker, topic, partition, port);
https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
788
In Kafka within a consumer group you can have max 1 consumer per partition.
Multi-Topic Consumers We may have a consumer group that listens to multiple topics. If they have the same key-partitioning scheme and number of partitions across two topics, we can join data across the two topics.
If there're more paritions than consumers in a group, some consumers will consume data from more than one partition. If there're more consumers in a group than paritions, some consumers will get no data. If you add new consumer instances to the group, they will take over some partitons from old members.
Reading records from partitions. Unlike the other pub/sub implementations, Kafka doesn't push messages to consumers. Instead, consumers have to pull messages off Kafka topic partitions. A consumer connects to a partition in a broker, reads the messages in the order in which they were written.
One instance of SimpleConsumer reads from a single partition. Though you can easily create multiple instances of SimpleConsumer and read different partitions sequentially or in parallel (from different threads).
The tricky part is coordination among readers on different machines so they don't read from the same partition (assuming all messages need to be processed only once). You need to use high level consumer or write similar custom code to accomplish that.
147
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
