Kafka Consumer Group: read new messages only

Question

I am using a kafka-node ConsumerGroup to read messages from Kafka. I want to read new messages from the topic, i.e. not all previously unconsumed messages but only messages that arrive after the creation of the ConsumerGroup.

The option fromOffset: 'latest'only comes into place when there is no stored offset for the given groupId. I can use this by creating a new groupId each time the ConsumerGroup is created, however I want to avoid this approach because this would create a large number of groupIds and stored offsets on the kafka server.

I also tried to manually set the offset to -1 like this:

offset = new kafka.Offset(consumer.client);
offset.commit(groupId, [{ topic: topic, partition: 0, offset: -1 }],
  function (err, data) {
    logger.debug("tried to set offset: ", data);

    if (err) logger.error("error", err);

  })
);

This returns an errorCode of 0 which should be a success, but the ConsumerGroup receives old messages afterwards.

Kafka version: 0.10.0 kafka-node: 1.6.2

Answer 1

Initiating the kafka ConsumerGroup with options:

 autoCommit: false,   
 fromOffset: 'latest'

creates a ConsumerGroup that will never commit an offset to Kafka (unless you do it manually, which is not wanted). Hence it will always chose the 'latest' option when it is started.

Answer 2

Just small update for all those who had noticed that even with 'autoCommit' set to 'false' the solution mentioned above doesn't work and offset is still stored for group: please be sure to set additional flag: commitOffsetsOnFirstJoin: false in option object. If flag is not set that offset will be stored per group and once started for the second time it'll get all messages starting from this offset

I hope it'll help to save (a lot of) time ;)