Running a script on an AWS server

Question

I have a script that I need to run once a day that requires a lot of memory. I would like to run it on a dedicated amazon box.

Is there some automated way to build a box, download all required software (like ruby) and then run my script. After the script is ran, I would like to shutdown the box.

The two options I can think of are:

1. I am thinking about hacking EMR to do this. (My script is a mapper against an empty directory)
2. Chef - This seemed like too much for one simple script.

Answer 1

You can accomplish setting up a new EC2 instance on startup using the official Ubuntu AMIs, the official Amazon Linux AMIs, and any other AMI that supports the concept of a user-data script.

Create a script (bash, Perl, Python,

1. whatever) that starts with #!
2. Pass this script as the user-data when running the EC2 instance.
3. The script will automatically be run as root on the first boot.

Here's the article where I introduced the concept of a user-data script:

Automate EC2 Instance Setup with user-data Scripts
http://alestic.com/2009/06/ec2-user-data-scripts

Your user-data script can install the required software, configure it, install your work script, and set up a cron job that runs the work script once a day.

ENHANCEMENT:

If the installation script don't take a long time to run (e.g., under an hour or few) then you don't even have to run a single dedicated instance 24 hours a day. You can instead use an approach that lets AWS start an instance for you on a regular schedule.

Here's an article I wrote that provides details on this approach with sample commands:

Running EC2 Instances on a Recurring Schedule with Auto Scaling
http://alestic.com/2011/11/ec2-schedule-instance

The general approach is to use Auto Scaling to start an instance with your user-data script on a regular schedule. Your job will terminate the instance when it has completed. They key is to suspend Auto Scaling's normal desire to re-start instances that terminate so that you don't pay for a running instance until the next time your job starts.