EC2 Autoscale Python script

Question

I want to run a python script on EC2. The python script needs to load a .pickle file. The script gets documents from the Amazon queue (SQS) and processes them.

My question: How does the autoscale really work? Are those separate instances (or virtual servers), who need to execute a version of the file, each separately? If that's the case, how do I execute the file on that instance? Create an AMI image which auto executes the script on startup? I found a way to just start instances with an image and send a script to run with the user_data paramater, but how does this work on autoscale? Like I said, are those seperate instances as well? Or is this just one instance which ads more cpu power and memory on scale up?

I'm creating a Python script which uses boto's autoscale interface (http://boto.readthedocs.org/en/latest/autoscale_tut.html) which I run in a cronjob. It needs to check the SQS multiple times a day and scale the instances up or down on the number of queue messages. Is that even the right way?

Answer 1

Erik, this sounds like you are on the right track. I think incorporating a few ideas from the built-in autoscaling process in EC2 may help you out here:

• Yes, I would build a master AMI that has everything loaded you need it to. Autoscaling usually works by scaling "out/in" (adding more/less instances) rather than scaling "up/down" (upsizing/downsizing the instance type). This helps give you more resiliency and no downtime for a reboot after a resize.
• As you've done with the user-data field, you could build into that AMI a bootstrapping script that runs particular updates or operations at warm-up time, OR simply build a retrieval script that will reach out to download a script (perhaps from S3) to then be executed. [If you use built-in IAM roles for security into your S3 bucket, you don't need to embed keys in your AMI for this download.]
• Provided that everything in your auto-scaling group is working behind a load balancer, you should then attach your new instances to the ELB (or your own HAproxy instance) when they are healthy.

So you are just basically emulating the workflow of AWS's auto-scaling, while also injecting your own bootstrapping:

Scale up:

• spawn new instance
• check for health
• attach to ELB
• check for HTTP health (or whatever other health check you set up) and let traffic pass into it.

Scale down:

• remove from ELB
• terminate instance

My preferred way to initiate the bootstrapping is to load a simple script to download a second shell script, using the user-data field. I normally do that with bash, but there's not a reason in the world it can't be a python script. But this way the bootstrapping script is isolated (and not stuck in the AMI), so I can update or tweak it freely without having to re-do the AMI.

A script built into the AMI /root/scripts/bootstrap-me.sh could simply contain:

#!/bin/bash 

/usr/bin/aws s3 cp s3://my.bucket/my-init-script.py ./  
/usr/bin/python my-init-script.py
/usr/bin/aws s3 cp s3://my.bucket/here-is-my.pickle ./

And then your my-init-script.py could contain whatever python-based logic you need it to, including execution of the .pickle file you've just grabbed.

Anytime you manually or automatically trigger autoscaling (to scale up) you should just pass the user-data to execute your bootstrap-me.sh shell script, and the new instance should be able to join the farm.