How to get last consumed offset for a consumer group?

Question

I have two consumers in a consumer group which have assigned same kafka topic partitions. I wish to get last read offset of say, consumer A from inside consumer B. Any Idea, how to implement this?

Answer 1

Kafka store offsets by (consumer-group-id, topic, partition) so the first thing to note is that from Kafka point of view there is no such thing like "last read offset of consumer A". All information that you can get with the Kafka consumer API is for a given (group, topic, partition). You have two methods in consumer API that may be useful.

commited(): Get the last committed offset for the given partition (whether the commit happened by this process or another).

position(): Get the offset of the next record that will be fetched (if a record with that offset exists).

If that is not what you need, then you will have to implement something yourself. Assuming you already know how to get the last offset read from consumer A, then consumer A should store that value in some location that is available to consumer B. This location could be

• Kafka itself. For example consumer A can publish last read offset to a well know topic like ConsumerA-p0 and Consumer B can subscribe to this topic.
• Zookeeper. Again, agreeing in a well known path.
• An external database.
• More rudimentary options if both consumers share the same OS: IPC, a file in the file system, a variable in memory protected with a lock, etc.

Answer 2

A single partition will never be assigned to two consumer instances in the same group.

You can use the below script to know the last consumed offset

sh kafka-consumer-groups.sh --bootstrap-server localhost:9092 --new-consumer --group groupname --describe