Menu
I have a topic which is read as GlobalKTable and Materialized in a store. The issue is if I update a key on the topic and then read from store, for a while(~0.5sec) I get the old value.
What could be the reason for this issue?
Is it that globalktable stores the data in rocksDB per application instance so if the key on another partition is updated it takes some time to pull data from all partitions and update its local rocksDB. If not, please explain how does globalktable store maintain its state internally?
How can the above issue be resolved ? Should we not use globalktable in such scenarios where consistency is expected to match that of say a mysql database?
364
Kafka can achieve around millisecond latency, by using synchronous messaging. With synchronous messaging, the producer does not collect messages into a patch before sending.
Kafka was traditionally used for high throughput rather than latency-sensitive messaging, but it does have a low-latency configuration. (Mostly setting linger.ms=0 and reducing buffer sizes). In this configuration, you can get below 1-millisecond latency a good percentage of the time for modest throughputs.
Only the Kafka Streams DSL has the notion of a GlobalKTable . Like a KTable, a GlobalKTable is an abstraction of a changelog stream, where each data record represents an update.
« Kafka Summit Americas 2021. The grace period is a parameter of windowed operations such as Window or Session aggregates, or stream-stream joins. This configuration determines how long after a window ends any new data will still be processed.
Is it that globalktable stores the data in rocksDB per application instance so if the key on another partition is updated it takes some time to pull data from all partitions and update its local rocksDB. If not, please explain how does globalktable store maintain its state internally?
Absolutely yes. There is always some latency until Kafka Streams poll() the topic again and updates is local RocksDB.
Should we not use globalktable in such scenarios where consistency is expected to match that of say a mysql database?
It depends on what guarantees you need -- if the producer writes into the GlobalKTable topic and the write was successful, this does not guarantee that a Kafka Streams application has consumed this write and has updated the GlobalKTable. Producers and Consumers are decoupled in Kafka by design.
200
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
