How to fire EC2 instances and upload/run a startup script on each of them?

Question

I want to automate the launch of a set of Linux EC2 instances.

Basically, I want to write a script/program that would :

• Instantiate N occurrences of a given AMI of mine.
• For each started instance, it would upload a customized script and let the script run into the instance.

Using VMWare, I would typically do that using vmrun or the Vix SDK.

What are the options in Amazon AWS/EC2?

Answer 1

The answer depends a bit on what AMI you are running as the features provided are entirely AMI dependent.

The Amazon Linux AMIS and the official Ubuntu AMIs have the cloud-init package installed. This has a number of ways you can trigger startup actions, but the one that matches your request most closely (and my favorite because I invented it) is the concept of a user-data script.

You can simply pass any script (starting with the two characters #!) as the user-data when starting the EC2 instances. It will be run as root on the first boot of the instance.

For a specific example of how this works, I use this exact technique in my recent article: Uploading Known ssh Host Key in EC2 user-data Script

You also wanted to run more than one EC2 instance with the same script. The ec2-run-instances command and the related APIs and web console allow you to specify any number of instances to start with the same user-data. For example:

ec2-run-instances            \   --instance-count 10        \   --user-data-file $MYSCRIPT \   --key $USER                \   $SOMEAMI 

If you are currently running an AMI that does not have cloud-init installed, you could do one of:

• Switch to an AMI that has cloud-init installed, or

• Build a custom version of your AMI that has cloud-init installed, or

• Write a more complicated wrapper script that makes a record of all of the instance ids after they are kicked off, waits for all of the instances to move to the running state, waits for the sshd to accept connections, uploads your startup script to each instance, and runs the startup script on each instance.