How to get metrics data from aws cloudwatch to csv

Question

i have been working on project, and i want to export metrics data from cloudwatch like CPU Utilization and Network Out data, is there any way to get that data? and convert it to csv?

Answer 1

Export CloudWatch Metrics

You can't do it directly but following is the step by step instructions:

Pre Reqs

1. AWS CLI (Command Line Interface) from here
2. jq is a lightweight and flexible command-line JSON processor from here

How to export

Use the following CLI:

aws cloudwatch get-metric-statistics 
--namespace AWS/EC2 --metric-name CPUUtilization 
--dimensions Name=InstanceId,Value=i-xxxxxxxxxxxxxxxxx 
--statistics Average 
--start-time 2020-03-01T00:00:00 
--end-time 2020-03-31T23:59:00 
--period 3600
--region us-east-1

Valid options for --metric-name depend on the --name-space parameter. For AWS/EC2, the full list can be seen by running the following CLI command:

aws cloudwatch list-metrics --namespace "AWS/EC2"

Valid options for --statistics are:

SampleCount
Average
Sum
Minimum
Maximum

--start-time and --end-time specify the range.

--period The granularity, in seconds, of the returned data points.

--region The region where the CloudWatch metric is being meatured (us-east-1, us-west-2 etc.)

Data output will look something like the following:

{
    "Label": "CPUUtilization",
    "Datapoints": []
}

To convert it to CSV we will use jq. For this you have two options:

Option 1

Pipe all the aws cli output to jq:

aws cloudwatch get-metric-statistics 
--namespace AWS/EC2 --metric-name CPUUtilization 
--dimensions Name=InstanceId,Value=i-xxxxxxxxxxxxxxxxx 
--statistics Average 
--start-time 2020-03-01T00:00:00 
--end-time 2020-03-31T23:59:00 
--period 3600
--region us-east-1
| jq -r '.Datapoints[] | [.Timestamp, .Minimum, .Unit] | @csv'

Option 2

Export the data to JSON:

aws cloudwatch get-metric-statistics 
--namespace AWS/EC2 --metric-name CPUUtilization 
--dimensions Name=InstanceId,Value=i-xxxxxxxxxxxxxxxxx 
--statistics Average 
--start-time 2020-03-01T00:00:00 
--end-time 2020-03-31T23:59:00 
--period 3600
--region us-east-1 >> data.json

Use jq to read the json to csv:

jq -r '.Datapoints[] | [.Timestamp, .Minimum, .Unit] | @csv' data.json

Output

Here is how the output will look like:

"2020-03-24T11:00:00Z",0.327868852454245,"Percent"
"2020-03-11T21:00:00Z",0.327868852454245,"Percent"
"2020-03-15T04:00:00Z",0.322580645156596,"Percent"
"2020-03-27T18:00:00Z",0.327868852478101,"Percent"

Answer 2

There is no in-built capability to export Amazon CloudWatch metrics to CSV.

There are API calls available to extract metrics, but you would need to write a program to call the API, receive the metrics and store it in a suitable format.

There are projects available that can assist with this, such as:

• mogproject/cloudwatch-dump: Just dump all the CloudWatch metrics.
• AWS – CloudWatch – DataExtractor – API | vMan
• romgapuz/awsmetric2csv: AWS Metric to CSV is a Python command-line utility to extract CloudWatch metric data from an AWS resources (e.g. EC2, RDS) and save to a CSV file.

If you are looking for these types of tools, make sure they refer to CloudWatch metrics, not CloudWatch Logs (which is something different).