Adding durability to in-memory data structures

Question

What are some of the popular techniques you can adopt to add durability to your in-memory data structures (ie) if the process crashes, you can preserve all previously executed operations on that data structure?

If my data structure involves just a list of tuples, then I would just store them in a SQL DB and that would give me durability for free. But what if my data structure was a graph or a tree?

The one thing I could think of is to explicitly log all operations to disk (append-only log) and in the event of a crash, replay the log to preserve the previous state. If the log becomes too big, then there will be a compaction step. I'm guessing this is what a database engine does internally for durability (checkpointing is what this process is called)?

Btw note that this is not a scenario where the entire dataset doesn't fit in memory.

Answer 1

You might want to try an object prevalence engine. For .NET, you might want to try Bamboo.Prevalence, which is a port of a similar engine called Prevayler for Java.

Answer 2

I've implemented the "Mrjb" technology in 2 companies' products, which is basically exactly what you've suggested in your question: a "Memory Resident Journal Backed" database, an in-memory data-structure where every change is logged to disk as it happens. And it works great for us!

http://www.edval.biz/memory-resident-programming-object-databases

I'd be happy to share our real-world experiences with using this in a production context. I love being able to replay an exact sequence of events or roll back to any point in time.