AWS Elastic Beanstalk Worker - Scale Based On Number Of Available Queue Messages

Question

I'm currently using AWS' Elastic Beanstalk workers for a queue of mine, the metrics available to trigger the autoscaling are pretty generic (CPU, Net in, Net out, etc).

I'm curious to know if it is possible to use a trigger based on the status of the queue attached to the worker - specifically adding or removing instances based on the average number of available messages in the queue over the last X minutes?

Answer 1

If you're using SQS this is very possible and recommended, If I remember correctly it was one of the examples in the Architecture course.

Heres further info on an example https://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/as-using-sqs-queue.html

Answer 2

Since this is the first hit for this kind of search I want to update with an answer. AWS has an example now of an .ebextensions config that does exactly what you want:

https://github.com/awsdocs/elastic-beanstalk-samples/blob/master/configuration-files/aws-provided/environment-configuration/workertier-scaleonqueuesize.config

Reproduced here:

Resources:
  AWSEBCloudwatchAlarmHigh:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmActions: []

  AWSEBCloudwatchAlarmLow:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmActions: []

  QueueDepthAlarmHigh:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmDescription: "Alarm if queue depth grows beyond 20 messages"
      Namespace: "AWS/SQS"
      MetricName: ApproximateNumberOfMessagesVisible
      Dimensions:
        - Name: QueueName
          Value: { "Fn::GetAtt": ["AWSEBWorkerQueue", "QueueName"] }
      Statistic: Sum
      Period: 300
      EvaluationPeriods: 1
      Threshold: 20
      ComparisonOperator: GreaterThanThreshold
      AlarmActions:
        - Ref: AWSEBAutoScalingScaleUpPolicy

  QueueDepthAlarmLow:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmDescription: "Alarm if queue depth is less than 5 messages"
      Namespace: "AWS/SQS"
      MetricName: ApproximateNumberOfMessagesVisible
      Dimensions:
        - Name: QueueName
          Value: { "Fn::GetAtt": ["AWSEBWorkerQueue", "QueueName"] }
      Statistic: Sum
      Period: 300
      EvaluationPeriods: 1
      Threshold: 5
      ComparisonOperator: LessThanThreshold
      AlarmActions:
        - Ref: AWSEBAutoScalingScaleDownPolicy

Basically it removes the default alarms and makes new ones based on message size. I've tested this configuration and it works as-is if you use the autogenerated queue from ElasticBeanstalk, but if you specify a pre-existing queue it won't. You can either hardcode the queue name in this script, or you can retrieve the URL and manipulate it to get the name. I'm not an expert with CloudFormation so there's probably a better way to do this but here's what I came up with:

Resources:

  # make the default alarms do nothing (I believe)
  AWSEBCloudwatchAlarmHigh:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmActions: []

  AWSEBCloudwatchAlarmLow:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmActions: []

  # set the High alarm
  QueueDepthAlarmHigh:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmDescription: "Alarm if queue depth grows beyond 50 messages"
      Namespace: "AWS/SQS"
      MetricName: ApproximateNumberOfMessagesVisible
      Dimensions:
        - Name: QueueName
          Value:
            "Fn::Select":
              - 4
              - "Fn::Split":
                - '/'
                - "Fn::GetOptionSetting":
                    Namespace: "aws:elasticbeanstalk:sqsd"
                    OptionName: "WorkerQueueURL"
      Statistic: Sum
      Period: 300
      EvaluationPeriods: 1
      Threshold: 50
      ComparisonOperator: GreaterThanThreshold
      AlarmActions:
        - Ref: AWSEBAutoScalingScaleUpPolicy

  # set the Low alarm
  QueueDepthAlarmLow:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmDescription: "Alarm if queue depth is less than 5 messages"
      Namespace: "AWS/SQS"
      MetricName: ApproximateNumberOfMessagesVisible
      Dimensions:
        - Name: QueueName
          Value:
            "Fn::Select":
              - 4
              - "Fn::Split":
                - '/'
                - "Fn::GetOptionSetting":
                    Namespace: "aws:elasticbeanstalk:sqsd"
                    OptionName: "WorkerQueueURL"
      Statistic: Sum
      Period: 300
      EvaluationPeriods: 1
      Threshold: 5
      ComparisonOperator: LessThanThreshold
      AlarmActions:
        - Ref: AWSEBAutoScalingScaleDownPolicy