How to run two console consumers in the same consumer group?

Question

When I run two instances of Kafka-console-consumers with the exact same properties (using the default one config/consumer.properties), I get same messages on both the instances.

./bin/kafka-console-consumer.sh --bootstrap-server :9092 --topic test1

If both the instances have the same consumer group id, shouldn't Kafka send a given message to only one of the consumers? How to run them as one consumer group?

Answer 1

From kafka docs i found this

The default for console consumer's enable.auto.commit property when no group.id is provided is now set to false. This is to avoid polluting the consumer coordinator cache as the auto-generated group is not likely to be used by other consumers.

But here is the trick, use this command to list all consumer groups across all topics, as you said i have opened four console consumers and i want to check list of consumer groups consuming from that topic

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

Every console consumer start with different group id, this is the reason always consuming from beginning addition of this property (--from-beginning)

ups.sh --bootstrap-server localhost:9092 --list
Note: This will not show information about old Zookeeper-based consumers.
console-consumer-66835
console-consumer-38647
console-consumer-18983
console-consumer-18365
console-consumer-96734

Okay easiest way to set group.id for console consumer

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer-property group.id=test1

Read up Managing Consumer Groups.

Answer 2

The trick is to use --consumer.config config/consumer.properties or --consumer-property group.id=test1 that would specify the group.id explicitly.

./bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic test1 \
  --consumer.config config/consumer.properties