Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

AWS Cloudwatch monitoring for S3

Amazon Cloudwatch provides some very useful metrics for monitoring my EC2s, load balancers, elasticache and RDS databases, etc and allows me to set alarms for a whole range of criteria; but is there any way to configure it to monitor my S3s as well? Or are there any other monitoring tools (besides simply enabling logging) that will help me monitor the numbers of POST/GET requests and data volumes for my S3 resources? And to provide alarms for thresholds of activity or increased datastorage?

like image 296
Mark Baker Avatar asked Nov 13 '13 09:11

Mark Baker


People also ask

What are Amazon CloudWatch metrics for Amazon S3?

Amazon CloudWatch metrics for Amazon S3 can help you understand and improve the performance of applications that use Amazon S3. There are several ways that you can use CloudWatch with Amazon S3.

How does monitor Amazon S3 work?

Monitor Amazon S3 requests to quickly identify and act on operational issues. The metrics are available at 1-minute intervals after some latency for processing. These CloudWatch metrics are billed at the same rate as the Amazon CloudWatch custom metrics.

How to monitor AWS S3 buckets?

Using CloudWatch to Monitor AWS S3 Buckets With the AWS CloudWatch support for S3 it is possible to get the size of each bucket, and the number of objects in it. Read on to see how you can use this to keep an eye on your S3 buckets to make sure your setup is running as expected.

How does CloudWatch work with AWS?

If an alarm is triggered, CloudWatch can take action automatically to enable Amazon EC2 Auto Scaling or stop an instance, for example, so you can automate capacity and resource planning. Monitor your applications that run on AWS (on Amazon EC2, containers, and serverless) or on-premises.


1 Answers

AWS S3 is a managed storage service. The only metrics available in AWS CloudWatch for S3 are NumberOfObjects and BucketSizeBytes. In order to understand your S3 usage better you need to do some extra work.

I have recently written an AWS Lambda function to do exactly what you ask for and it's available here:

https://github.com/maginetv/s3logs-cloudwatch

It works by parsing S3 Server side log files and aggregates/exports metrics to AWS Cloudwatch (CloudWatch allows you to publish custom metrics).

Example graphs that you will get in AWS CloudWatch after deploying this function on your AWS account are:

RestGetObject_RequestCount
RestPutObject_RequestCount
RestHeadObject_RequestCount
BatchDeleteObject_RequestCount
RestPostMultiObjectDelete_RequestCount
RestGetObject_HTTP_2XX_RequestCount
RestGetObject_HTTP_4XX_RequestCount
RestGetObject_HTTP_5XX_RequestCount
+ many others

Since metrics are exported to CloudWatch, you can easily set up alarms for them as well. CloudFormation template is included in GitHub repo and you can deploy this function very quickly to gain visibility into your S3 bucket usage.

EDIT 2016-12-10:

In November 2016 AWS has added extra S3 request metrics in CloudWatch that can be enabled when needed. This includes metrics like AllRequests, GetRequests, PutRequests, DeleteRequests, HeadRequests etc. See Monitoring Metrics with Amazon CloudWatch documentation for more details about this feature.

like image 170
Michal Gasek Avatar answered Sep 21 '22 13:09

Michal Gasek