Menu
When a Kafka Consumer fails to deserialize a message, is it the client applications responsibility to deal with the Poison Message?
Or
Does Kafka "increment" the message offset and continue consumption of valid messages?
Is there a "Best Practice" for dealing with Poison Messages held on Kafka topics?
467
ERROR HANDLING IN CONSUMER Consumer is trying to consume data from Kafka topic but the connection to broker is not established because broker is not available. In that case, consumer should retry to consume data within some time intervals. Kafka records are stored in the topics.
Consumers work in groups to read data from Kafka topics. Consumers within a group are responsible for reading a portion of the messages (partitions) so that messages can be consumed in parallel. Consumers "pull" messages from Kafka. Kafka just sits there while consumer applications simultaneously read the same data.
How to Ensure the Order of Messages. In Kafka, order can only be guaranteed within a partition. This means that if messages were sent from the producer in a specific order, the broker will write them to a partition and all consumers will read from that in the same order.
Increasing the number of partitions and the number of brokers in a cluster will lead to increased parallelism of message consumption, which in turn improves the throughput of a Kafka cluster; however, the time required to replicate data across replica sets will also increase.
When Kafka is unable to deserialize the record the consumer will receive a org.apache.kafka.common.KafkaException, you should commit the offset yourself and keep consuming.
89
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
