cosmosdb - archive data older than n years into cold storage

Question

I researched several places and could not find any direction on what options are there to archive old data from cosmosdb into a cold storage. I see for DynamoDb in AWS it is mentioned that you can move dynamodb data into S3. But not sure what options are for cosmosdb. I understand there is time to live option where the data will be deleted after certain date but I am interested in archiving versus deleting. Any direction would be greatly appreciated. Thanks

Answer 1

I don't think there is a single-click built-in feature in CosmosDB to achieve that.

Still, as you mentioned appreciating any directions, then I suggest you consider DocumentDB Data Migration Tool.

Notes about Data Migration Tool:

• you can specify a query to extract only the cold-data (for example, by creation date stored within documents).
• supports exporting export to various targets (JSON file, blob storage, DB, another cosmosDB collection, etc..),
• compacts the data in the process - can merge documents into single array document and zip it.
• Once you have the configuration set up you can script this to be triggered automatically using your favorite scheduling tool.
• you can easily reverse the source and target to restore the cold data to active store (or to dev, test, backup, etc).

To remove exported data you could use the mentioned TTL feature, but that could cause data loss should your export step fail. I would suggest writing and executing a Stored Procedure to query and delete all exported documents with single call. That SP would not execute automatically but could be included in the automation script and executed only if data was exported successfully first. See: Azure Cosmos DB server-side programming: Stored procedures, database triggers, and UDFs.

UPDATE: These days CosmosDB has added Change feed. this really simplifies writing a carbon copy somewhere else.