Scaling Down Auto Scaling Group in AWS causes perpetual alarm

Question

I am currently using two scaling policies which are attached to my auto scaling group: A

1. A scale up policy which is invoked when a CloudWatch alarm is invoked. This CloudWatch alarm uses the CPUUtilization metric and fires at CPU over 80%.
2. The other is a scale down policy which is invoked when a different CloudWatch alarm is invoked. This CloudWatch alarm uses the CPUUtilization metric and fires when the CPU is under 50%.

The side effect of this approach is that when my ASG instances are idle (fully scaled down, no processing occurring) my ASG is in an alarm state.

Is there a way to set this up differently so that my ASG is not in a state of constant alarm?

Below is a segment of these alarms from my CloudFormation template:

"ScaleUpPolicy" : {
  "Type" : "AWS::AutoScaling::ScalingPolicy",
  "Properties" : {
    "AdjustmentType" : "ChangeInCapacity",
    "AutoScalingGroupName" : { "Ref" : "WebApplicationASG" },
    "Cooldown" : "1",
    "ScalingAdjustment" : "1"
  }
},
"CPUAlarmHigh": {
  "Type": "AWS::CloudWatch::Alarm",
  "Properties": {
    "EvaluationPeriods": "1",
    "Statistic": "Average",
    "Threshold": "80",
    "AlarmDescription": "Alarm if CPU too high or metric disappears indicating instance is down",
    "Period": "60",
    "AlarmActions": [ { "Ref": "ScaleUpPolicy" } ],
    "Namespace": "AWS/EC2",
    "Dimensions": [ {
      "Name": "AutoScalingGroupName",
      "Value": { "Ref": "WebApplicationASG" }
    } ],
    "ComparisonOperator": "GreaterThanThreshold",
    "MetricName": "CPUUtilization"
  }
},
"ScaleDownPolicy" : {
  "Type" : "AWS::AutoScaling::ScalingPolicy",
  "Properties" : {
    "AdjustmentType" : "ChangeInCapacity",
    "AutoScalingGroupName" : { "Ref" : "WebApplicationASG" },
    "Cooldown" : "1",
    "ScalingAdjustment" : "-1"
  }
},
"CPUAlarmLow": {
  "Type": "AWS::CloudWatch::Alarm",
  "Properties": {
    "EvaluationPeriods": "1",
    "Statistic": "Average",
    "Threshold": "50",
    "AlarmDescription": "Alarm if CPU is low, causing scale down",
    "Period": "60",
    "AlarmActions": [ { "Ref": "ScaleDownPolicy" } ],
    "Namespace": "AWS/EC2",
    "Dimensions": [ {
      "Name": "AutoScalingGroupName",
      "Value": { "Ref": "WebApplicationASG" }
    } ],
    "ComparisonOperator": "LessThanThreshold",
    "MetricName": "CPUUtilization"
  }
},

Answer 1

You can hide these in the Console, by clicking the "Hide Autoscaling Alarms" checkbox.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/hide-autoscaling-alarms.html

Still not ideal, but better than nothing.

Answer 2

This is the normal and expected behavior.

Having a metric in the alarm state is not a problem - remember that it is the change of an alarm state that triggers events. So presumably once your scale up trigger goes into alarm, the scale down would come out of alarm. Then when the metric goes down, it goes back into an alarm state, and a scale down event is triggered.