EC2 automation tools / strategies?

Question

What tools or strategies are you using for automation of EC2 activities?

I need to be able to bring up a number of EC2 instances, provision various software to it (primarily Python packages), interact with S3 (primarily download data), and run various jobs. I'll be doing this both on-demand and on a scheduled basis.

I'm trying to decide if I should:

• Create an AMI with all my software loaded on it or
• Launch a plain vanilla linux AMI instance and scp my software to it

For the provisioning and automation Boto looks pretty good. Or I could write something with Paramiko. Recommend either or anything else I should be looking it?

Basically I'm looking for advice / success stories, let me know what's working for you.

Answer 1

To answer your bullets about selecting AMIs, I would say that it depends on how much software you're installing.

I have been successful with a hybrid approach, where I build an AMI and load my heavyweight and more stable software. This is the stuff that needs to run an installer, or takes considerable time to install (remember that if you re-install a package every time as part of your startup process, you're paying for the install every time). Then, I upload the small and volatile software at provisioning/startup time. In this bucket goes most of the application code, data, etc. That way, I can change my app and not have to touch the AMI.

The benefits of this approach:

• Don't have to pay for running the same software install thousands of times.
• AMI can stay fairly stable over time.
• Can use software that requires intervention or GUI interaction to install.

Major drawbacks:

• Your AMI's OS version will become stale over time.
• Your AMI may not be flexible as to the instance type/architecture it will run on. For instance, you may create it on a 32-bit OS and thereby prevent it from running on the High CPU instance types, or vice versa. So you may lock yourself into a pricing scheme.

I don't use Python, so I can't comment on either of the APIs you referenced.