What is a recommended scalable DB platform to use in AWS for large amounts of volatile data sets - elasticsearch, Redis or DynamoDB?

Question

Users of our platform will have large amounts of stored data on our system. Through an application, once connected, that data will be transferred to them and no longer need to remain on our servers. There could potentially be hundreds or thousands of users connected at any given time, performing their downloads.

Here's the proposed architecture:

User management, configuration, and data download statistics will be maintained in a SQL Server database, while using either Redis or DynamoDB for the large data sets.

The reason for choosing either Redis or DynamoDB is based on cost - cheaper than running another SQL Server instance, and performance. The data format will be similar to a datamart - flat table with no joins.

Initially the queries would be simple - get all data for user X between a date range, and optionally delete.

Since we may want to add free text searching for certain fields of that data using elasticsearch may be a better option to use from the get-go.

I want this to be auto-scaling but not sure which database would be best to use for this scenario.

Answer 1

Here's some great discussion on Database + Search tier from AWS ReInvent:

https://youtu.be/K7o5OlRLtvU?t=1574