Given you have multiple systems, which are integrated by events, and all of them are using event sourcing. Where do you store the events?
In my case I have three systems:
Whenever a domain event happens in one of those systems the event is published and can be processed by the other systems. All systems are using event sourcing.
I am wondering where you would save the events. Of course each system has to store all events that it processed because it is using event sourcing and therefore depends on the events it once processed.
But what about the other events that where not needed and therefore the system did not subscribe to? I am struggling with the fact that requirements can change, such that a system would have to process events from the past that it did not persist. Where would you get these events from, if the system needed to process events that it did not subscribe when they occured?
I think there is a big difference to systems that do not use event sourcing at this point. If you have to implement a feature in a system A which depends on data, that is not available in A, but in another system B, and you persistent current state via a ORM tool like NHibernate you can simply import that data from A to B. Since a system, that uses event sourcing, depends on events to get to it's current state you have to import all the events that you missed in the past but are need now.
For me there are a few different approaches to this problem.
How do you handle such a situation? Where do you save your events?
Edit
Thanks Roy Dictus for your answer. I'm still not sure how to handle the following situation:
The website publishes the events CustomerRegistered, CustomerPurchasedProduct and CustomerMarkedProductAsFavorite. In the current version of the backend customers haave to be displayed and their purchases have to be displayed. What a customer marked as a favorite is not of interest in that version of the system. Thus the backend only subscribed to CustomerRegistered and CustomerPurchasedProduct.
Now the marketing department also wants the information about the favorite products to be shown on the customer details page. Since the backend didn't subscribe to CustomerMarkedProductAsFavorite this information is not available in the backend. Where do I get that information from?
Instead of saving latest status of data into database, Event Sourcing pattern offers to save all events into database with sequential ordered of data events. This events database called event store. Instead of updating the status of a data record, it append each change to a sequential list of events.
On the technical level, event sourcing can be implemented using dedicated storage systems, as well as general-purpose "NoSQL" and SQL databases. If you're interested in the origins of event sourcing, the articles by Greg Young on event sourcing and CQRS are a great place to start.
TL;DR: Kafka is not an event store; rather, it is an enabler for building event stores. For the most trivial of use cases, Kafka's support for unbounded retention, combined with per-entity record keying, tombstones and topic compaction may be used to build a very rudimentary event store.
EDIT to accommodate your extra question:
If another application suddenly becomes interested in extra information, it has to add listeners to the events it is now interested in.
Furthermore, all sources of these events can then replay those events. Replay is a powerful feature of event-driven systems that allows for such scenarios. So, the event sources replay only the selected events (say, all CustomerMarkedItemAsFavorite events of the last 6 months). Systems that have already consumed these events should recognize that the events replayed are "old" ones (i.e., ones that it has already processed) and ignore them.
This way, any subsystem that is updated to use extra information from the other subsystems can get that information and get all up-to-date in a single batch operation.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With