Can we archive the AWS QLDB data, as every business have limitations on amount of history that can be stored

Question

I was looking at AWS QLDB service to store the audit trail history of changes that were made to our application, so that it can be immutable.

But, at the end it is a database & we can't just keep on adding data (storing such large amount of data is costly).

At some point in time we will need to roll over / archive existing data & start over everything a fresh.

Was wondering how AWS QLDB will be able to handle such scenarios ?

P.S. I am a newbie to AWS QLDB.

Answer 1

as every business have limitations on amount of history that can be stored

I would argue that this assertion too broad.

There are business cases that need to be able to document historical events indefinitely (or indefinitely, for all practical purposes) because either the data remains relevant or because without retaining the entire history, there is no way to conclusively prove that the current state of the database is as it should be... and that is the purpose of QLDB -- maintaining historical records that cannot be modified or deleted, either accidentally or on purpose.

With QLDB, your data’s change history is immutable – it cannot be altered or deleted – and using cryptography, you can easily verify that there have been no unintended modifications to your application’s data.

https://aws.amazon.com/qldb/

Each transaction builds on the one before it. Oversimplified, it lools like this:

hash(t1) = SHA256(t1)
hash(t2) = SHA256(t2 + hash(t1))
hash(t3) = SHA256(t3 + hash(t2))
...

Those hash values are also stored, so each transaction can be cryptographically verified against its predecessor, all the way back to the beginning of time. Deleting older records removes information necessary for verifying newer records.

A use case where you plan to purge historical data seems like an incorrect application of QLDB.