Process count on remote machines on AWS

Question

I am designing an Auto Scaling system for my application which runs on Amazon EC2 instances. The application reads messages from SQS and processes them.

The Auto Scaling system will monitor two things:

1. The number of messages in the SQS,
2. The total number of processes running on all EC2 machines.

For example, if the number of messages in SQS exceeds 3000 I want the system to autoscale, create a new EC2 instance, deploy code on it and whenever the number of messages goes below 2000 i want the system to terminate an EC2 instance.

I am doing this with Ruby and capistrano. My question is:

I am unable to find a method to determine number of processes running on all EC2 machines and save the number inside a variable. Could you help me?

Answer 1

You might want to utilize cron and CloudWatch API to push the numbers manually to CloudWatch as a part of auto-scaling-group policy. By numbers I mean the number of processes from each instance ps aux | grep your_process | wc -l

CloudWatch will let you set alarm for that manual metric aggregated by SUM of the nr of processes across either all running instances or by auto-scaling-group.

Something to let you get started:

Pushing RAM Memory metrics manually: http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/mon-scripts-perl.html

One more: http://aws.typepad.com/aws/2011/05/amazon-cloudwatch-user-defined-metrics.html

For memory it looks simple, as amazon already provides scripts for this. For processes you might need to dig in these scripts or read the official API docs

EDIT:

If you are now worried about single-point-of-failure in the watching system and you have a list of servers it might be preferred to examine them in parallel from a remote server:

rm ~/count.log

# SSH in parallel
for ROW in `cat ~/ListofIP.txt`
do
    IP=`echo ${ROW} | sed 's/\./ /g' | awk '{print $1}'`
    ssh -i /path/to/keyfile root@${IP} "ps -ef | grep process_name.rb | grep -v grep | wc -l" >> ~/count.log &
done

# Wait for totals
while [ ! `wc -l ~/ListofIP.txt` -eq `wc -l ~/count.log` ]
do
  wait 1
done

# Sum up numbers from ~/count.log
# Push TO CloudWatch