Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How can I get the group.id of a topic in command line in Kafka?

Tags:

apache-kafka

I installed kafka on my server and want to learn how to use it, I found a sample code written by scala, below is part of it,

def createConsumerConfig(zookeeper: String, groupId: String): ConsumerConfig = {
    val props = new Properties()
    props.put("zookeeper.connect", zookeeper)
    props.put("group.id", groupId)
    props.put("auto.offset.reset", "largest")
    props.put("zookeeper.session.timeout.ms", "400")
    props.put("zookeeper.sync.time.ms", "200")
    props.put("auto.commit.interval.ms", "1000")
    val config = new ConsumerConfig(props)
    config
}

but I don't know how to find the group id on my server.

like image 552
Frank Yeung Avatar asked Dec 28 '15 11:12

Frank Yeung


People also ask

How do I find my Kafka group ID?

Step1: Open the Windows command prompt. Step2: Use the '-group' command as: 'kafka-console-consumer -bootstrap-server localhost:9092 -topic -group <group_name>' . Give some name to the group. Press enter.

How do you get list of consumer groups in Kafka for a topic?

Get the list of consumer groups for a topic. Use kafka-consumer-groups.sh to list all consumer groups. Note that the below command will list all the consumer groups for all topics managed by the cluster.

What is group name in Kafka?

This offset is stored based on the name provided to Kafka when the process starts. This name is referred to as the Consumer Group. The Consumer Group name is global across a Kafka cluster, so you should be careful that any 'old' logic Consumers be shutdown before starting new code.

What is group ID in Kafka producer?

group.id specifies the name of the consumer group a Kafka consumer belongs to. When the Kafka consumer is constructed and group.id does not exist yet (i.e. there are no existing consumers that are part of the group), the consumer group will be created automatically. Note.


Video Answer


3 Answers

The group id is something you define yourself for your consumer by providing a string id for it. All consumers started with the same id will "cooperate" and read topics in a coordinated way where each consumer instance will handle a subset of the messages in a topic. Providing a non-existent group id will be considered to be a new consumer and create a new entry in Zookeeper where committed offsets will be stored.

like image 111
Lundahl Avatar answered Nov 15 '22 10:11

Lundahl


You could get a Zookeeper shell and list path where Kafka stores consumers' offsets like this:

./bin/zookeeper-shell.sh localhost:2181
ls /consumers

You'll get a list of all groups.

EDIT: I missed the part where you said that you're setting this up yourself so I thought that you want to list the consumer groups of an existing cluster.
Lundahl is right, this is a property that you define, which is used to coordinate consumer threads so that they don't consume "each other's" messages (each consumes a subset). If you, for example, use 2 consumers with different groups, they'll each consume the whole topic.

like image 21
Marko Bonaci Avatar answered Nov 15 '22 11:11

Marko Bonaci


/kafkadir/kafka-consumer-groups.sh --all-topics --bootstrap-server hostname:port --list

like image 27
Eduardo A EDUARDO Fernandez Di Avatar answered Nov 15 '22 11:11

Eduardo A EDUARDO Fernandez Di