Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Multiple availability zones with terraform on AWS

The VPC I'm working on has 3 logical tiers: Web, App and DB. For each tier there is one subnet in each availability zone. Total of 6 subnets in the region I'm using.

I'm trying to create EC2 instances using a module and the count parameter but I don't know how to tell terraform to use the two subnets of the App tier. An additional constraint I have is to use static IP addresses (or a way to have a deterministic private name)

I'm playing around with the resource

resource "aws_instance" "app_server" {
  ...
  count = "${var.app_servers_count}"

  # Not all at the same time, though!
  availability_zone = ...
  subnet_id = ...
  private_ip = ...
}

Things I've tried/thought so far:

  • Use data "aws_subnet" "all_app_subnets" {...}, filter by name, get all the subnets that match and use them as a list. But aws_subnet cannot return a list;
  • Use data "aws_availability_zones" {...} to find all the zones. But I still have the problem of assigning the correct subnet;
  • Use data "aws_subnet_ids" {...} which looks like the best option. But apparently it doesn't have a filter option to match the networks namel
  • Pass the subnets IDs as list of strings to the module. But I don't want to hard code the IDs, it's not automation;
  • Hard code the subnets as data "aws_subnet" "app_subnet_1" {...}, data "aws_subnet" "app_subnet_2" {...} but then I have to use separate sets of variables for each subnet which I don't like;
  • Get information for each subnet like in the point above but then create a map to access it as a list. But it's not possibile to use interpolation in variables definition;
  • Not using modules and hard-code each instance for each environment. Mmmm... really?

I really ran out of ideas. It seems that nobody has to deploy instances in specific subnetworks and keep a good degree of abstration. I see only examples where subnetworks are not specified or where people just use default values for everything. Is this really something so unusual?

Thanks in advance to everyone.

like image 578
ColOfAbRiX Avatar asked Sep 04 '17 16:09

ColOfAbRiX


People also ask

Can an EC2 instance be in multiple availability zones?

Amazon EC2 is hosted in multiple locations world-wide. These locations are composed of AWS Regions, Availability Zones, Local Zones, AWS Outposts, and Wavelength Zones. Each Region is a separate geographic area. Availability Zones are multiple, isolated locations within each Region.

What is multi availability Zone in AWS?

Each AWS Region is subdivided into separate Availability Zones. Each Availability Zone has its own power, cooling, and network connectivity and thus forms an isolated failure domain. Within the constructs of AWS, customers are encouraged to run their workloads in more than one Availability Zone.

Can one availability Zone have multiple subnets?

After creating a VPC, you can add one or more subnets in each Availability Zone. Each subnet must reside exclusively within one Availability Zone and cannot span zones. AWS assigns a unique ID to each subnet.

How many availability zones an instance can belong to?

There are anywhere between two and five availability zones in an AWS Region. Moving forward, the standard will be three or more per region.


4 Answers

It is possible to evenly distribute instances across multiple zones using modulo.

variable "zone" {
  description = "for single zone deployment"
  default = "europe-west4-b"
}

variable "zones" {
  description = "for multi zone deployment"
  default = ["europe-west4-b", "europe-west4-c"]
}

resource "google_compute_instance" "default" {
  count = "${var.role.count}"
  ...
  zone = "${var.zone != "" ? var.zone: var.zones[ count.index % length(var.zones) ]}"
  ...
}

This distribution mechanism allow to distribute nodes evenly across zones.
E.g. zones = [A,B] - instance-1 will be in A, instance-2 will in B, instance-3 will be in A again.
By adding zone C to zones will shift instance-3 to C.

like image 106
Tom Lime Avatar answered Oct 13 '22 15:10

Tom Lime


The count index in the resource will throw an error if you have more instances than subnets. Use the element interpolation from Terraform

element(list, index) - Returns a single element from a list at the given index. If the index is greater than the number of elements, this function will wrap using a standard mod algorithm. This function only works on flat lists.

subnet_id = "${element(data.aws_subnet_ids.app_tier_ids.ids, count.index)}"
like image 23
RayKeck Avatar answered Oct 13 '22 14:10

RayKeck


At the end I figured out how to do it, using data "aws_subnet_ids" {...} and more importantly understanding that terraform creates lists out of resources when using count:

variable "target_vpc" {}
variable "app_server_count" {}
variable "app_server_ip_start" {}

# Discover VPC
data "aws_vpc" "target_vpc" {
  filter = {
    name = "tag:Name"
    values = [var.target_vpc]
  }
}

# Discover subnet IDs. This requires the subnetworks to be tagged with Tier = "AppTier"
data "aws_subnet_ids" "app_tier_ids" {
  vpc_id = data.aws_vpc.target_vpc.id
  tags {
    Tier = "AppTier"
  }
}

# Discover subnets and create a list, one for each found ID
data "aws_subnet" "app_tier" {
  count = length(data.aws_subnet_ids.app_tier_ids.ids)
  id = data.aws_subnet_ids.app_tier_ids.ids[count.index]
}

resource "aws_instance" "app_server" {
  ...

  # Create N instances
  count = var.app_server_count

  # Use the "count.index" subnet
  subnet_id = data.aws_subnet_ids.app_tier_ids.ids[count.index]

  # Create an IP address using the CIDR of the subnet
  private_ip = cidrhost(element(data.aws_subnet.app_tier.*.cidr_block, count.index), var.app_server_ip_start + count.index)

  ...
}
like image 4
ColOfAbRiX Avatar answered Oct 13 '22 13:10

ColOfAbRiX


I get Terraform to loop through the subnets in an availability zone by using the aws_subnet_ids data source and filtering by a tag representing the tier (in my case public/private).

This then looks something like this:

variable "vpc" {}
variable "ami" {}
variable "subnet_tier" {}
variable "instance_count" {}

data "aws_vpc" "selected" {
  tags {
    Name = "${var.vpc}"
  }
}

data "aws_subnet_ids" "selected" {
  vpc_id = "${data.aws_vpc.selected.id}"

  tags {
    Tier = "${var.subnet_tier}"
  }
}

resource "aws_instance" "instance" {
  count         = "${var.instance_count}"
  ami           = "${var.ami}"
  subnet_id     = "${data.aws_subnet_ids.selected.ids[count.index]}"
  instance_type = "${var.instance_type}"
}

This returns a consistent sort order but not necessarily starting with AZ A in your account. I suspect that the AWS API returns the subnets in AZ order but ordered by their own internal id as the AZs are shuffled by account (presumably to stop AZ A being flooded as humans are predictably bad at putting everything in the first place they can use).

You would have to tie yourself in some horrible knots if for some odd reason you particularly care about instances being placed in AZ A first but this minimal example should at least get instances being round-robined through the AZs you have subnets in by relying on Terraform's looping back through arrays when exceeding the array length.

like image 3
ydaetskcoR Avatar answered Oct 13 '22 13:10

ydaetskcoR