Handling large amount of events in event sourcing

Question

CQRS with event sourcing looks like a perfect fit as an architecture for one of our systems, there is only one little thing we are current worried about: Handling a large amount of events and dealing with huge event stores as a consequence.

Our current system receives about a million events a day (which currently have nothing to do with event sourcing though), if we were to store them all over a longer period of time, our event stores might get pretty big but if we dump/purge to a rolling snapshot frequently, we might loose one of the big advantages of event sourcing: information about the history of the system and replay.

What are common ways to deal with this problem in a CQRS architecture? Is it a problem at all? Do we just throw more hardware at the event store or is there something we can do at the architecture design level?

Answer 1

I think the most common approach is to use snapshots and persistent read models. That is, you don't actually replay your events very often, except when you need to build a new read model or change the way an existing one works. By storing snapshots of your domain objects, you avoid having to replay long streams of events.

One could argue that storing snapshots and persistent read models isn't a whole lot different than just doing CQRS without event-sourcing. But the old events are there in the event that you made a mistake in your read model, or need to derive new information, or have other strict auditing requirements.

In our application, where we have many events that have low business value, we plan to scrub events heavily during execution so that our event logs stay smaller. But I imagine for some objects we will still fall back to snapshots and persistent models.