Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

understanding consumer group id

I did fresh installation of Apache Kafka 0.10.1.0.

I was able to send / receive messages on command prompt.

While using Producer / Consumer Java Example, I am not able to know group.id parameter on Consumer Example.

Let me know on how to fix this issue.

Below is Consumer Example I had used:

public static void main(String[] args) {              Properties props = new Properties();              props.put("bootstrap.servers", "localhost:9092");              props.put("group.id", "my-topic");              props.put("enable.auto.commit", "true");              props.put("auto.commit.interval.ms", "1000");              props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");              props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");              KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);              try {                  consumer.subscribe(Arrays.asList("my-topic"));                       ConsumerRecords<String, String> records = consumer.poll(100);                      System.err.println("records size=>"+records.count());                      for (ConsumerRecord<String, String> record : records)                           System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());                  }              catch (Exception ex){                  ex.printStackTrace();              }             finally {                  consumer.close();             }         } 

After running the command for consumer, I can see the messages (on the console) posted by producer. But unable to see the messages from java program

bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic my-topic --from-beginning

like image 887
Ankit Avatar asked Dec 29 '16 09:12

Ankit


People also ask

What is consumer groupId?

consumer.group id is to load balance the produced data (if the group.id is different for each consumer, each consumer will get the copy of data) if partition=1 and total consumers count = 2, only one out of two active consumer will get data.

How does Kafka group.id work?

Kafka consumers belonging to the same consumer group share a group id. The consumers in a group then divides the topic partitions as fairly amongst themselves as possible by establishing that each partition is only consumed by a single consumer from the group.

How do consumer groups work?

Every consumer inside a consumer group is assigned a number of a topic's partitions, ensuring that records are shared between the group while also keeping each partition's records in order. This mechanism is made more efficient by keeping the assigned partitions the same until introducing a new consumer to the group.

Is group.id mandatory for Kafka consumer?

The consumer group-id is mandatory, it plays a major role when it comes to scalable message consumption. To start a consumer group-id is mandatory.


1 Answers

Consumers label themselves with a consumer group name, and each record published to a topic is delivered to one consumer instance within each subscribing consumer group. Consumer instances can be in separate processes or on separate machines.

If all the consumer instances have the same consumer group, then the records will effectively be load balanced over the consumer instances.

If all the consumer instances have different consumer groups, then each record will be broadcast to all the consumer processes.

The group.id is a string that uniquely identifies the group of consumer processes to which this consumer belongs.

(Kafka intro)

like image 60
Raz Omessi Avatar answered Sep 23 '22 08:09

Raz Omessi