Read Kafka message and set that message is processing by client

Question

Are there any feature in Kafka that client can set e.g. flag or some atribute for message?

User story for this is: Client reads message from Kafka and set flag or atribute for message which logical means "message processing". Client do sth with information e.g. sending to other system. If sending to other system was succesfull then client sets in Kafka's message that message was processed. After that message then shouldn't be get/use by other consumer, message processing was done.

Is it posible with any built in mechanism in kafka?

Answer 1

It is possible in Kafka but, there is no concept like marking the message as being processed

A message can be committed once it is processed, so that no other consumers in the group read that message again (until you manually seek).

If the processing is a heavy operation, you can divide it into steps and write the intermediate results in intermediate topics.

If you are running multiple consumers as a group each of the consumers will get a subset of data (called topic partitions) to work on. So, no consumer belonging to the group interferes with another consumer in its group.

However, in case if one of the consumers in the group dies, any of the other consumers takeover its work from the point where it left off (see Transactional processing) and process them.

You might be interested in having exactly-once processing semantics where a message is processed only once.