Menu
I just configured Cloudwatch logs on my ec2 instances and am loving it so far. I also set up alerts for certain keywords, like "ERROR". While the email alert seems to be working fine, I was wondering if there's a way to fine-tune the alert email to make it a little concise & informative. Specifically, I'm looking to
Get rid of all the boilerplate text in the alert email.
Include some information about the Error/Exception that triggered the alert. This could be something as simple as including the log statement that generated the alert.
Right now, the alert email looks like
You are receiving this email because your Amazon CloudWatch Alarm "App-Error-Alarm" in the US East - N. Virginia region has entered the ALARM state, because "Threshold Crossed: 1 datapoint (1.0) was greater than or equal to the threshold (1.0)." at "Tuesday 07 February, 2017 16:39:43 UTC".
View this alarm in the AWS Management Console: https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#s=Alarms&alarm=App-Error-Alarm
Alarm Details: - Name: App-Error-Alarm - Description: Errors in app.log - State Change: INSUFFICIENT_DATA -> ALARM - Reason for State Change: Threshold Crossed: 1 datapoint (1.0) was greater than or equal to the threshold (1.0). - Timestamp: Tuesday 07 February, 2017 16:39:43 UTC - AWS Account: <>
Threshold: - The alarm is in the ALARM state when the metric is GreaterThanOrEqualToThreshold 1.0 for 300 seconds.
Monitored Metric: - MetricNamespace: LogMetrics - MetricName: ERROR - Dimensions: - Period: 300 seconds - Statistic: Sum - Unit: not specified
State Change Actions: - OK: - ALARM: [arn:aws:sns:us-east-1:<>:support] - INSUFFICIENT_DATA:
I'd like it to something like
Alarm: App-Error-Alarm
Keyword: "ERROR"
Reason: ERROR 2017-02-07 07:31:47,375 [SimpleAsyncTaskExecutor-5] com.app.server.rest.Watcher: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
Its short, sweet and instantly tells me whether its something that needs my immediate attention. Can this be done without writing code as suggested here?
654
Login to the AWS console and navigate to the CloudWatch Service. Once you're in the CloudWatch console go to Logs in the menu and then highlight the CloudTrail log group. After that you can click the “Create Metric Filter” button. In the “Filter Pattern” box we'll select a pattern that we're looking for.
Open the CloudWatch console, choose Rules, and then choose Create rule. For Event source, choose Event pattern, and then choose Custom event pattern. To match on specific alarm state transitions, add the ARNs of your alarms.
Via the CloudWatch logs, find the log group for the function, and click Search Events. Set the date/time to a little bit before the error occurred. Enter a keyword you think will appear in the error.
You have this problem because you configured an alarm and is meant for aggregated data, not for specific log record. You configure it for some metric (number of log records with ERROR keyword).
You can use log subscription instead and stream all log records matching a filter to a custom Lambda function. You can use it to send notifications to email or Slack.
To configure log streaming, go to Lambda in AWS console and create a new function from a blueprint named "cloudwatch-logs-process-data". It has a basic structure and is easy to customize to your needs.
62
You can customize Cloudwatch alarms by using Cloudwatch Logs Customize alarms Package provided by AWS labs.
When you get an alarm, you want enough information to decide whether it needs immediate attention or not. You also want to customize the alarm text to operational needs. The CloudWatch Logs Customize Alarms is a Lambda function that helps in reading the logs from CloudWatch Logs during an alarm and send a customized email through SES.
The packages provides for using a Lambda SNS endpoint, which can then customize the alarm notification according to your operational requirements.
31
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
