Creating multiple instances with for_each

Question

I'd love some help with Terraform's count/for_each functions.

The goal is to read multiple json files(current two) into a list of maps and create specific amount of aws_instances with specific naming convention.

Configurations

cat test_service_1.json
{
      "instance_name": "front",
      "instance_count": "3",
      "instance_type": "t2.micro",
      "subnet_type": "private",
      "elb": "yes",
      "data_volume": ["no", "0"]
}

cat test_service_2.json
{
      "instance_name": "back",
      "instance_count": "3",
      "instance_type": "t2.micro",
      "subnet_type": "private",
      "elb": "yes",
      "data_volume": ["no", "0"]
}

cat main.tf
locals {
  services = [jsondecode(file("${path.module}/test_service_1.json")),
  jsondecode(file("${path.module}/test_service_2.json"))]
}

resource "aws_instance" "test_instance" {
  ami           = "amzn-ami-hvm-2018.03.0.20200206.0-x86_64-gp2"
  instance_type = "t2.micro"
  tags = merge(
    map("Name", "prod-app-?"),
    map("env", "prod")
  )
}

Eventually I want the code to go over both json files and create:

prod-front-1
prod-front-2
prod-front-3
prod-back-1
prod-back-2
prod-back-3

I can do that with [count.index +1] but I don't know how to loop through more than one map.

Answer 1

When using resource for_each our task is always to write an expression that produces a map where there is one element per instance we want to create. In this case, that seems to be an expression that can expand a single object containing a count to instead be multiple objects of the number given in the count.

The building blocks we can use to do this in Terraform are:

• for expressions to project one collection value into another.
• The range function to generate sequences of integers given a count.
• The flatten function to turn multiple nested lists into a single flat list.

Let's take this step by step. We'll start with your existing expression to load the data from the JSON files:

locals {
  services = [
    jsondecode(file("${path.module}/test_service_1.json")),
    jsondecode(file("${path.module}/test_service_2.json")),
  ]
}

The result of this expression is a list of objects, one per file. Next, we'll expand each of those objects into a list of objects whose length is given in instance_count:

locals {
  service_instance_groups = [
    for svc in local.services : [
      for i in range(1, svc.instance_count+1) : {
        instance_name = "${svc.instance_name}-${i}"
        instance_type = svc.instance_type
        subnet_type   = svc.subnet_type
        elb           = svc.elb
        data_volume   = svc.data_volume
      }
    ]
  ]
}

The result of this one is a list of lists of objects, each of which will have a unique instance_name value due to concatenating the value i to the end.

To use for_each though we we will need a flat collection with one element per instance, so we'll use the flatten function to achieve that:

locals {
  service_instances = flatten(local.service_instance_groups)
}

Now we have a list of objects again, but with six elements (three from each of the two input objects) instead of two.

Finally, we need to project that list to be a map whose keys are the unique identifiers Terraform will use to track the instances. I usually prefer to do this final step directly inside the for_each argument because this result is specific to that use-case and unlikely to be used anywhere else in the module:

resource "aws_instance" "test_instance" {
  for_each = {
    for inst in local.service_instances : inst.instance_name => inst
  }

  ami           = "amzn-ami-hvm-2018.03.0.20200206.0-x86_64-gp2"
  instance_type = each.value.instance_type
  tags = {
    Name = "prod-app-${each.key}"
    Env  = "prod"
  }
}

This should result in Terraform planning to create instances with addresses like aws_instance.test_instance["front-2"].

I wrote each of the above steps out separately to explain what each one was achieving, but in practice I'd usually do the service_instance_groups and service_instances steps together in a single expression, because that intermediate service_instance_groups result isn't likely to be reused elsewhere. Bringing that all together into a single example, then:

locals {
  services = [
    jsondecode(file("${path.module}/test_service_1.json")),
    jsondecode(file("${path.module}/test_service_2.json")),
  ]
  service_instances = flatten([
    for svc in local.services : [
      for i in range(1, svc.instance_count+1) : {
        instance_name = "${svc.instance_name}-${i}"
        instance_type = svc.instance_type
        subnet_type   = svc.subnet_type
        elb           = svc.elb
        data_volume   = svc.data_volume
      }
    ]
  ])
}

resource "aws_instance" "test_instance" {
  for_each = {
    for inst in local.service_instances : inst.instance_name => inst
  }

  ami           = "amzn-ami-hvm-2018.03.0.20200206.0-x86_64-gp2"
  instance_type = each.value.instance_type
  tags = {
    Name = "prod-app-${each.key}"
    Env  = "prod"
  }
}

---

As a bonus, beyond what you were asking about here, if you give those JSON files systematic names and group them together into a subdirectory then you could use Terraform's fileset function to to automatically pick up any new files added in that directory later, without changing the Terraform configuration. For example:

locals {
  services = [
    for fn in fileset("${path.module}", "services/*.json") :
    jsondecode(file("${path.module}/${fn}"))
  ]
}

The above will produce a list containing an object for each of the files in the services subdirectory that have names ending in .json.