Will the messages get deleted even if it's not consumed in Kafka

Question

In Kafka, will there be any situation such that the log retention policy is configured in a way that it will actually delete the messages even before the consumer can read that message?

Answer 1

Yes that is possible.

Kafka does not track how many times each message is consumed. After a message has reached its retention limits (due to time or size), it will be eligible to deletion.

In practice, deletion only happens on segments so messages stay longer than the limits but you should not rely on that.

Note that consumers expose a records-lead metrics that indicates how far they are from the front (where messages are deleted) of the log. If you see this metric dropping over time, it is an indicator you may run into this scenario.