Menu
we plan to use Kafka as a central component in our data warehouse given that the producer is able to handle transactions (in short: rollbacks and commits).
When googling Kafka + Transactions I find a lot of theoretical thoughts about the possibility of how Kafka could handle transactions but at the moment I do not see any function in the java API that supports commits and rollbacks for the producer.
Has anybody made some experiences with transactions and Kafka and can give me an hint?
914
Transactions in Kafka are designed so that they are mainly handled on the producer/message broker side, rather than the consumer side. The consumer is effectively an idempotent reader, while the producer/coordinator handle the transaction.
The transaction log is an internal kafka topic. Each coordinator owns some subset of the partitions in the transaction log, ie. the partitions for which its broker is the leader. Every transactional.id is mapped to a specific partition of the transaction log through a simple hashing function.
Any attempt to involve any third-party data sources in the business process is stuck because the Kafka fundamentally does not support XA transactions.
Conclusion: Kafka can handle the volume With 5 brokers it makes autoscaling the Kafka cluster to 4 vCPU and 12 GB memory we are able to sustain 200 million events per hour or 55,000 events per second simple.
I think what you are looking for is basically called transactional messaging in Kafka where producers are capable of creating session (aka transactional session) and send messages within the sessions. Hence it can choose to either commit / abort the transaction.
[Source]: Please read the wiki for details
55
No; Kafka does not support transactions.
You can get certainty that a message has been produced to a partition, but once produced you are not able to rollback that message.
Since version 0.11.0 Apache Kafka supports transactions: https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka
23
Actually, from the last version 0.11.0.0 transactions are supported. See Guarantee unique global transaction for Kafka Producers
3
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
