Menu
what is a role of leader epoch checkpoint in log segment of Kafka partition?
900
This is only available for the active segment (log file). leader-epoch-checkpoint - It refers to the number of leaders previously assigned by the controller. The replicas use the leader epoch as a means of verifying the current leader. The leader-epoch-checkpoint file contains two columns: epochs and offsets.
The partition leader is a Kafka Broker. Partition Leader. This is clearly mentioned in Kafka Docs: Each partition has one server which acts as the "leader" and zero or more servers which act as "followers".
log file is an actual segment containing records up to a specific offset. The name of the file defines the starting offset of the records in that log. The . index file contains an index that maps a logical offset (in effect the record's id) to the byte offset of the record within the . log file.
The leader handles all the read and writes operations of data for the partitions. The leader and its followers are determined by the zookeeper(discussed later). If the broker holding the leader for the partition fails to serve the data due to any failure, one of its respective ISR replicas will takeover the leadership.
In Kafka, a leader epoch refers to the number of leaders previously assigned by the controller. Every time a leader fails, the controller selects the new leader, increments the current "leader epoch" by 1, and shares the leader epoch with all replicas. The replicas use the leader epoch as a means of verifying the current leader. If a leader fails and returns, when it tries to contact other replicas, it will send what it believes is the current leader epoch. The replicas will ignore the messages sent with outdated leader epochs.
The leader-epoch-checkpoint file contains two columns: epochs and offsets, as shown here. Each row is a checkpoint for the latest recorded leader epoch and the leader's latest offset upon becoming leader. Both replicas and leaders contain this file. Its role is for checking what range of offsets pertain to which epoch.
61
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
