EC2 instance placement

Question

We have a lot of m1.medium/large instances running in ec2. Generally these machines best suit our needs bang/buck-wize. However, we are getting some really horrible throughput between the nodes. A bit of digging shows that these nodes are on different subnets and possibly in different data centers.

To get more throughput I would like to try get EC2 to put these nodes closer together (network-wize) if at all possible. Our topology is as follows:

• A "server" node hosting some central stored data on EBS, NFS exported, always on. m1.large.
• A couple of primary worker nodes that are on 80% of the time. m1-large
• 0-50 Slave nodes that are only ever on for a few hours at a time. generally these are m1.smalls or m1.medium but can be m1.large if required by the job.

Slave nodes mount the exported NFS drives.

Obviously, if the slaves get spun up in a place far away from the server, we get some horrible throughput.

I am aware of placement groups, however these seem to require much more expensive instance types and a completely different visualization technology and the possibility that when we need 40 extra nodes there isn't room for them in our placement group.

This would greatly increase our costs and reduce our flexibility.

Any suggestions?

Answer 1

You need to launch your instances and placement groups in the same availability zone:

http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html

Availability zones can be chosen when the instance is launched or placement group is created (ec2-create-placement-group). The zones are usually either A,B,C or No Preference (default - random).

Nodes in the same availability zone are connected on the same 10Gbps network - thats better than the bandwidth on a solid-state drive running on a SATA3 Interface - 6Gbps.

The different zones are located in the same region (e.g. North East USA), but are usually a fair distance away to allow for disaster recovery implementations. They're definitely in different datacenters, and are probably in different states in the same region eg North East USA. Amazon don't publish the actual locations.

Note, there doesn't seem to be a way to specify an availability zone for a placement group creation through the AWS Management Console - you'll probably need to use the API.

http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-CreatePlacementGroup.html

I've seen entire availability zones go down on EC2 a few times - so if you're going to move your entire infrastructure to the same availability zone you're putting your application at risk, you need some kind of disaster recovery or fault tolerant implementation between 2 zones.

Hope this helps.