Idempotent and Transactions

Question

I am exploring Transactions in Kafka, and I want to understand all the details.

I noticed in Spring-Kafka that idempotent is enabled when you provide a transactionsalId.

public void setTransactionIdPrefix(String transactionIdPrefix) {
    Assert.notNull(transactionIdPrefix, "'transactionIdPrefix' cannot be null");
    this.transactionIdPrefix = transactionIdPrefix;
    enableIdempotentBehaviour();
}

At first glance, I assume Spring-Kafka enabled idempotent in transactions because it is "good-to-have". I assumed it was to ensure to ensure exactly-once semantics in transactions.

I did a bit more digging and discovered that idempotent is required for transactions to work. This is mentioned in KIP-98

Note that enable.idempotence must be enabled if a TransactionalId is configured.

Kafka idempotent is a feature to avoid duplicated messages, such as network errors after the message has been sent.

My understanding is that, Kafka transactions basically writes to an internal topic and idempotent has to be enabled to avoid duplicates.

Idempotent enables exactly-once semantics for producers.

Transactions enables exactly-once semantics for transitivity; consume -> produce.

Is my understanding correct?

What enables exactly-once for only consumers? Committing offset, idempotent, or transactions.

Answer 1

The Idempotent producer enables exactly once for a producer against a single topic. Basically each single message send has stonger guarantees and will not be duplicated in case there's an error.

The Transactional producer on the other hand enables to group a number of send (that can be across many partitions) together and have all of them (or none) applied. Transactions can also contain offset commits (in the end commiting offsets is the same as writing to a topic).

Because Consumers fetch data from Kafka, it's sort of already exactly once. When the consumer asks Kafka messages from offset N, if it does not receives them, it will just retry, there can't be any duplication. The only exactly once need for COnsumers is for committing offsets and that can be done by the Transactional Producer (The consumer needs to pass its current offsets to the Producer).