Using AWS for application logging database

Question

I'm currently working on a large web application that produces a large amount of log data. Because we don't have the infrastructure to log all the events to a database, we are writing them to file. Unfortunately, this makes it very difficult to search the logs for a specific event and impossible to generate reports on frequency.

While trying to figure out how to implement better database logging, I found Amazon's services. Specifically SimpleDB and DynamoDB. One of the use cases for SimpleDB was logging, but then later it states that

Amazon SimpleDB is designed to store relatively small amounts of data...

This seems contradictory. Here are my questions:

1. Would these database services be suitable for logging application events?
2. Would the be suitable for generating reports from log data?
3. Would I use a Timestamp as my primary key?
4. Are there drawbacks to this sort of service or something else I should consider?

Update 2018-06-13: I have since used SimpleDB to log application data on large applications. The key was to partition the logs into domains corresponding to the time period they were generated (daily for example) to ensure they didn't grow beyond their limit. Then set up a CRON job to periodically delete the old domains. This solution has worked well and is easily searchable.

Answer 1

My answer is based off of my experience with SimpleDB in a production environment.

1. I log thousands of application events a day to SimpleDB.
2. We have researchers that regularly pull the log data from SimpleDB and they've never complained. I'm not sure about running report queries on the SimpleDB itself, but I don't see why this would be a problem, when I run queries they are always fast for me.
3. SimpleDB indexes all columns, there's no "primary key".
4. I personally don't enjoy using the third-party tools that have been built to interact with SimpleDB. Also, consider the price based on how much data and processing you'll be utilizing.

Answer 2

SimpleDB is designed to store relatively small amounts of data. If you have "a large amount of log data" and you want to "search the logs for a specific event" and "generate reports on frequency", then you definitely want to go with DynamoDB.

DynamoDB has both "hash keys" (e.g., indexes) and "range keys" which allow you to sort the data in various ways. For example: "Show me all of the INFO logs from last week until now."

Also, if you check out the official DynamoDB forum, the developers and support staff are really good about following up and making sure your questions are answered.