AWS AutoScaling, downscale - wait for processes termination

Question

I want to use AWS AutoScaling to scaledown a group of instances when SQS queue is short. These instances do some heavy work that sometimes requires 5-10 minutes to complete. And I want this work to be completed before the instance termination.

I know a lot of people should have faced the same problem. Is it possible on EC2 to handle the AWS termination request and complete all my running processes before the instance is actually terminated? What is the best approach to this?

Answer 1

You could also use Lifecycle hooks. You would need a way to control a specific worker remotely, because AWS will select a particular instance to put in Terminating:Wait state and you need to manage that instance. You would want to take the following actions:

1. instruct the worker process running on the instance to not accept any more work.
2. wait for the worker to finish the work it already is handling
3. call the complete-lifecycle action.

AWS will take care of the rest for you.

ps. if you are using celery to power your workers then you can remotely ask a worker to shutdown gracefully. It won't shutdown unless it finishes with the tasks it had started executing.

Answer 2

Assuming you are using linux, you can create a pre-baked AMI that you use in your Launch Config attached to your Auto Scaling Group.

In the AMI you can put a script under /etc/init.d say /etc/init.d/servicesdown. This script would execute anything that you need to shutdown which would be scripts under /usr/share/services for example.

Here's kind like the gist:

servicesdown

It would always get executed when doing a graceful shutdown.

Then say on Ubuntu/Debian you would do something like this to add it to your shutdown sequence:

/usr/sbin/update-rc.d servicesdown stop 25 0 1 6 .

On CentOS/RedHat you can use the chkconfig command to add it to the right shutdown runlevel.