Menu
I am considering using Apache Kafka as a distributed message publisher to many subscribers. It is the perfect fit for me, since the solution has to scale easily.
The Kafka's documentation states that the message may be acknowledged thus ensuring the message delivery. However, today I came across this article which states that there are scenarios in which the messages may be lost. Then again, the article is only available in Google cache, so I do not know whether it is trustworthy...
So I have one doubt - is there any moment, any scenario, in which the message will be lost? In another words - my main requirement is that each message must reach its destination. Can it be met by using the Apache Kafka? Is it the right tool for this job?
723
So what are Kafka's delivery guarantees, and how does Kafka work behind the scenes to guarantee this behavior? Kafka uses a producer-consumer pattern to work with streaming data. Some processes are producers responsible for sending messages, and others are consumers responsible for receiving and processing them.
Apache Kafka supports 3 types of message delivery guarantees: at most once, at least once, exactly once. It is important to choose what guarantee we need from the beginning because this choice will influence the configuration of our producers and consumers and also the performance that Kafka can provide.
Kafka is primarily used to build real-time streaming data pipelines and applications that adapt to the data streams. It combines messaging, storage, and stream processing to allow storage and analysis of both historical and real-time data.
Kafka uses a binary protocol over TCP. The protocol defines all APIs as request response message pairs. All messages are size delimited and are made up of the following primitive types.
The original of the article you are looking for is here: http://engineering.onlive.com/2013/12/12/didnt-use-kafka/
If you read the full article and the comments you'll see much of the concern is not about the guarantee of at least once delivery, but that it was delivered AND successfully processed by the client. The last couple of comments on the article, including by the original author, seem to indicate he's satisfied with the approach.
You might also find this article of interest - similar concerns:
https://www.mail-archive.com/users%40kafka.apache.org/msg04492.html
And from some of the documentation:
So effectively Kafka guarantees at-least-once delivery by default and allows the user to implement at most once delivery by disabling retries on the producer and committing its offset prior to processing a batch of messages. Exactly-once delivery requires co-operation with the destination storage system but Kafka provides the offset which makes implementing this straight-forward.
Most of the conversations I've seen are not about the guarantee of at least once, but how to go from there to at most once or to exactly once.
69
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
