Apache Kafka - Resetting the last seen epoch of partition. Why?

Question

We use Kafka Mirror Maker Version 1 to mirror data between Kafka Clusters. I know that MM1 is deprecated but it is a solid piece of software that does exactly what we need. We use it out of a dedicated Kafka installation that is independent of the clusters between which we store data (which are running Kafka Version 2.6 now).

We used Kafka 2.7.x for the "MirrorMaker-Kafka" and recently updated this one to 3.3.1. Since then we have a lot of the following messages in out MM log:

2022-12-01 15:07:45,368 INFO Metadata - [Producer clientId=*****] Resetting the last seen epoch of partition MyTopic-5 to 87 since the associated topicId changed from null to c59OzubzRAO-yhA72TSFEw
2022-12-01 15:12:45,373 INFO Metadata - [Producer clientId=*****] Resetting the last seen epoch of partition MyTopic-5 to 87 since the associated topicId changed from null to c59OzubzRAO-yhA72TSFEw
2022-12-01 15:22:45,388 INFO Metadata - [Producer clientId=*****] Resetting the last seen epoch of partition MyTopic-5 to 87 since the associated topicId changed from null to c59OzubzRAO-yhA72TSFEw
2022-12-01 15:37:45,394 INFO Metadata - [Producer clientId=*****] Resetting the last seen epoch of partition MyTopic-5 to 87 since the associated topicId changed from null to c59OzubzRAO-yhA72TSFEw
2022-12-01 15:42:45,398 INFO Metadata - [Producer clientId=*****] Resetting the last seen epoch of partition MyTopic-5 to 87 since the associated topicId changed from null to c59OzubzRAO-yhA72TSFEw

As you can see the message is repeated every 5 minutes. We mirror several hundred partitions and the message is logged for many (maybe all) partitions.

As I could found out the message was introduced in the KAFKA-12257 fix in this commit.

Unfortunately, the meaning of the message is not clear to me. The constant repetition at a certain interval also makes me wonder. Possibly I still have weaknesses in my (Producer) configuration.

If someone could explain the phenomenon and knows what measures I can take to improve it, I would be very grateful.

Answer 1

according to https://forum.confluent.io/t/resetting-the-last-seen-epoch-of-partition/6740/2:

When a client first try to consume/produce from/to a topic, it will initialise an object in memory to track the topic metadata, and upon the first metadata response received from the kafka cluster.It will update its state from the default null to the actual topic ID.

that makes it sound like this error can be safely ignored, as long as it is changing from null, rather than some other value, in which case i would look at it in more detail.

Answer 2

The message that you are seeing is related to the KAFKA-12257 fix, which was implemented to address an issue with the producer client not properly tracking the epoch of a topic partition. Specifically, the fix ensures that the producer client is able to keep track of the epoch of a partition by associating a unique identifier (the "topicId") with it.

In your case, the message is being logged every 5 minutes because the producer client is resetting the last seen epoch of a specific partition (MyTopic-5) to 87, since the associated topicId changed from null to c59OzubzRAO-yhA72TSFEw.

To ensure that the producer is able to properly track the epoch of a partition, you should review your producer configuration and ensure that the "topicId" is set properly. Additionally, you should also ensure that your producer client is configured to properly handle partition reassignment events, as this can cause the topicId to change.

Finally, if you are still having issues with the producer client not properly tracking the epoch of a partition, you may want to consider upgrading to a newer version of Kafka Mirror Maker, as MM1 is now deprecated.