Menu
This question is similar to Using Kafka as a (CQRS) Eventstore. Good idea?, but more implementation specific. How to use kafka as event store, when I have thousands of event "sources" (aggregate roots in DDD)? As I've read in linked question and some other places, I'll have problems with topic per source. If I split events to topics by type, it will be much easier to consume and store, but I need access to event stream of particular source. How to do event sourcing with kafka?
391
Something like Citus seems appropriate for scalability, partitioning by tentant+stream. Kafka is still very useful in distributed scenarios. It is a non-trivial problem to expose each service's key events to other services. An event store is typically not built for that, but it's precisely what Kafka does well.
The problem with Kafka is that it only guarantees the order within partitions, not cross-partition, which leaves you with solving the ordering problem in some other way. And again, now you need to add complexity to solve a problem that you only have because you wanted to have a jack-of-all-trades service.
Kafka as an Event Bus. Apache Kafka is a distributed event streaming platform, originally developed by LinkedIn and open sourced since 2011. It is used by a vast number of companies to build high-performance data pipelines, enable real-time data analysis, and integrate data from critical applications.
Post all of your event sources to a single topic with a data type (thrift?) that includes some unique identifier for each event source. Then create consumers for each event type that you are interested in and identify each with a unique consumer group name. This way each unique source consumer will have its own offset value in zookeeper. Everybody reads the whole topic but only outputs (or deals with) info from a single source (or group of sources).
192
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
