Append to a log file in S3 (or any other AWS service)

Question

I need to create a log file in AWS S3 (or any other AWS service that can help here). AFAIU there is no way to append a line to an existing log file in S3. This means that I would either need to retrieve and resend the whole log each time a new message comes, or that I will need to create a new object per message. The latter option is complicated when retrieving data.

I have lots of log messages and it is important not to lose them so it is not an option to buffer them in my server memory and send to S3 once in a while.

Which AWS service would be my best option (also in terms of saving costs).

Answer 1

What you are looking for is called EBS (Elastic Block Store).

You can attach an EBS drive to any server, and start logging there. If your server dies, the EBS disk will still be around and you can look at the data later.

Even better, just create a central syslog server (with an EBS drive), and have all your (stateless) app servers send their logs there.

When your EBS drive fills up, archive the day/week/etc into S3 logs. This lets you store fairly large files in S3 (and even do compression if you want).

Answer 2

AWS SimpleDB would do what you want:

http://aws.amazon.com/simpledb/usecases_logging/

Since Amazon SimpleDB allows you to completely offload the work required to run a production database, many developers find it an ideal, low-touch data store for logging information about conditions or events, status updates, recurring activities, workflow processes, or device and application states. Amazon SimpleDB lets you cost-effectively “set and forget” these data logs and use them for diverse purposes, such as:

Monitoring or tracking Metering Trend of business analysis Auditing Archival or regulation compliance Application examples include:

Storing server logs centrally to reduce the space they consume on each running server Logging operational metrics or the results of ongoing performance tests for later analysis Auditing access entries or configuration changes for applications or networked devices Capturing and monitoring environment conditions (temperature, pressure levels, humidity, etc.) at various locations and programming alerts for particular conditions Logging and tracking geolocation information about objects or process status for activities in a workflow Multiple attributes of Amazon SimpleDB make it an attractive data store for data logs:

Central, with High Availability – If your data logs were previously being trapped locally in multiple devices/objects, applications, or process silos, you’ll enjoy the benefit of being able to access your data centrally in one place in the cloud. What’s more, Amazon SimpleDB automatically and geo-redundantly replicates your data to ensure high availability. This means that unlike a centralized on-premise solution, you’re not creating a single point of failure with Amazon SimpleDB, and your data will be there when you need it. All of the data can be stored via web services requests with one solution and then accessed by any device. Zero Administration – You store your data items with simple web services requests and Amazon Web Services takes care of the rest. The set it and forget it nature of the service means you aren’t spending time on database management in order to store and maintain data logs.

Cost-efficient – Amazon SimpleDB charges inexpensive prices to store and query your data logs. Since you are paying as you go for only the resources you consume, you don’t need to do your own capacity planning or worry about database load. The service simply responds to request volume as it comes and goes, charging you only for the actual resources consumed. To learn more about Amazon Si