Menu
We're moving to s3 to start serving some of the statically generated content for our web app. We have been looking at a mechanism for building a metric system about the usage of our site and we were planning on parsing the access logs for S3 by passing additional information to be logged on the content GET requests. We happened across the following entry in the developers guide:
Best Effort Server Log Delivery
The server access logging feature is designed for best effort. You can expect that most requests against a bucket that is properly configured for logging will result in a delivered log record, and that most log records will be delivered within a few hours of the time that they were recorded.
However, the server logging feature is offered on a best-effort basis. The completeness and timeliness of server logging is not guaranteed. The log record for a particular request might be delivered long after the request was actually processed, or it might not be delivered at all. The purpose of server logs is to give the bucket owner an idea of the nature of traffic against his or her bucket. It is not meant to be a complete accounting of all requests.
We are wondering what other people have experienced with respect to the delivery of access logs? Our alternative is to build an HTTP server and try to meter the metrics ourselves with a different call, but we think that parsing the log files could prove to be less work. We'd like to know if people have seen situations where delivery didn't take place to try to gauge about how accurate we could hope to be because some of the metrics we gather are used in some of our business processes.
800
Amazon S3 provides access to easy-to-use management tools that helps developers organize data as well as configure it to meet specific business compliance requirements more efficiently. S3 also has a 99.99999999999 durability rate (famously known as the 11 9's), which makes it highly reliable and secure.
Its random, as in not regular intervals. I have seen 5 per hour and sometimes single lines per file on low traffic buckets and 5 per minute from high traffic buckets with many lines per file. You have to aggregate your logs by yourself anyway.
You can record the actions that are taken by users, roles, or AWS services on Amazon S3 resources and maintain log records for auditing and compliance purposes. To do this, you can use server access logging, AWS CloudTrail logging, or a combination of both.
Server access logging provides detailed records for the requests that are made to an Amazon S3 bucket. Server access logs are useful for many applications. For example, access log information can be useful in security and access audits.
I was surprised how large my log files on S3 had gotten in under a month. It wasn't necessary for my app to parse the logs on Amazon but I like your approach. From what I've seen, you can expect the log files to be accurate and complete. Based on their CYA warning, the logs should not be used for anything critical.
124
We have been using S3 to log relatively large amounts of data (in the order of 100M rows). We needed to rely on S3 access logs for a specific purpose and we are observing few issues that might be important for potential users of access logs to be aware of:
My recommendation is to avoid relying on S3 access logs if data accuracy and completeness is critical.
27
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
