Terraform: Creating and validating multiple ACM certificates

Question

I'm running into a really confusing Terraform resource issue automating the generation and DNS validation of SSL certificates in ACM for a list of (Terraform-managed) hosted zones. Code can also be found in this gist.

I'm starting by bootstrapping hosted zones referencing this environment-specific variable.

hosted_zones = [
    {
        domain = "site1.com"
        zone_id = "MANUALLY FILL"
    }
]

The block I am using to build the zones seems to work reliably.

resource "aws_route53_zone" "zones" {
    count = "${length(var.hosted_zones)}"
    name  = "${lookup(var.hosted_zones[count.index], "domain")}"
}

After the zones are built, I am manually copying the zone ID into the variable because I haven't come up with a clever way to automate it given a combination of limitations of HCL and my lack of experience with it.

I can reliably generate naked and splat certificates for each hosted zone using...

resource "aws_acm_certificate" "cert" {
    count = "${length(var.hosted_zones)}"
    domain_name = "${lookup(var.hosted_zones[count.index], "domain")}"
    subject_alternative_names = ["*.${lookup(var.hosted_zones[count.index], "domain")}"]
    validation_method = "DNS"

    tags {
        Project = "${var.project}"
        Environment = "${var.environment}"
    }
}

Where things get hairy is when I try to automate the DNS validation for the certs. There is a good example in the documentation for a single hosted zone, but I haven't been able to successfully port it to multiple hosted-zones. My attempt...

resource "aws_route53_record" "cert_validation" {
    count = "${length(var.hosted_zones)}"

    name = "${aws_acm_certificate.cert.*.domain_validation_options.0.resource_record_name[count.index]}"
    type = "${aws_acm_certificate.cert.*.domain_validation_options.0.resource_record_type[count.index]}"
    zone_id = "${var.zone_override != "" ? var.zone_override : lookup(var.hosted_zones[count.index], "zone_id")}"
    records = ["${aws_acm_certificate.cert.*.domain_validation_options.0.resource_record_value[count.index]}"]
    ttl = 60
}

resource "aws_acm_certificate_validation" "cert" {
    count = "${length(var.hosted_zones)}"

    certificate_arn = "${aws_acm_certificate.cert.*.arn[count.index]}"
    validation_record_fqdns = ["${aws_route53_record.cert_validation.*.fqdn[count.index]}"]
}

The error I am seeing on first run is:

* module.acm.aws_route53_record.cert_validation: 1 error(s) occurred:
* module.acm.aws_route53_record.cert_validation: Resource 'aws_acm_certificate.cert' does not have attribute 'domain_validation_options.0.resource_record_value' for variable 'aws_acm_certificate.cert.*.domain_validation_options.0.resource_record_value'

The obnoxious part is that if I comment the validation resources, the apply succeeds, and then uncommenting them and re-running also succeeds.

I've tried (what feels like) every permutation of element() lookup(), list() and map() to target certificates by index in the output from the first resource block, but am running into documented "flat list" limitations and this is the closest I've gotten to success. I'd like to understand why the workaround is necessary so I can eliminate it. This feels like a syntax issue or me trying to get HCL to behave more like an OO language than it is.

Thank you for any experience that may help!

Answer 1

I had a similar scenario and the key to solving it was the use of locals and flatten(). The approach should also work for you such that you shouldn't need two passes to create the resources.

In this scenario there are multiple domains that each have subdomains that will appear in the subjectAltName section of the certificate. For example:

├── preview.example.com
│   ├── app.preview.example.com
│   └── www.preview.example.com
├── demo.example.com
│   ├── app.demo.example.com
│   └── www.demo.example.com
├── staging.example.com
│   ├── app.staging.example.com
│   └── www.staging.example.com
└── example.com
    ├── app.example.com
    └── www.example.com

To achieve this we first set some variables:

variable "domains" {
    type = "list"
    default = [
        "demo.example.com",
        "preview.example.com",
        "staging.example.com",
        "example.com"
    ]
}
variable "subdomains" {
    type = "list"
    default = [
        "app",
        "www"
    ]
}

Next we create the certificate resources that contain the subdomains as SANs.

resource "aws_acm_certificate" "cert" {
  count             = "${length(var.domains)}"
  domain_name       = "${element(var.domains, count.index)}"
  validation_method = "DNS"

  subject_alternative_names = ["${
    formatlist("%s.%s",
      var.subdomains,
      element(var.domains, count.index)
    )
  }"]
}

Next we're going to need a local variable to flatten the resulting set of domains and subdomains. This is needed because terraform doesn't support nested list syntax as of version 0.11.7, neither via the element() interpolation nor the `list[count].

locals {
  dvo = "${flatten(aws_acm_certificate.cert.*.domain_validation_options)}"
}

We'll next need a lookup of the Route 53 zone that we can use in the subsequent Route 53 records:

data "aws_route53_zone" "zone" {
  count        = "${length(var.domains) > 0 ? 1 : 0}"
  name         = "example.com."
  private_zone = false
}

We then create the Route 53 DNS records that will be populated with data from the certificate resource for DNS validation. We're adding one to the subdomains so that we also have a record for the base domain not included in the list of subdomains.

resource "aws_route53_record" "cert_validation" {
  count   = "${length(var.domains) * (length(var.subdomains) + 1)}"
  zone_id = "${data.aws_route53_zone.zone.id}"
  ttl     = 60

  name    = "${lookup(local.dvo[count.index], "resource_record_name")}"
  type    = "${lookup(local.dvo[count.index], "resource_record_type")}"
  records = ["${lookup(local.dvo[count.index], "resource_record_value")}"]
}

Finally we create the certificate validation resource that will wait for the certificate to be issued.

resource "aws_acm_certificate_validation" "cert" {
  count                   = "${length(var.domains) * (length(var.subdomains) + 1)}"
  certificate_arn         = "${element(aws_acm_certificate.cert.*.arn, count.index)}"
  validation_record_fqdns = ["${aws_route53_record.cert_validation.*.fqdn}"]
}

The one caveat for this last resource is that it'll create one instance of the resource for every certificate requested, but each instance will depend on all the FQDNs across all domains and subdomains. This won't affect anything in AWS but the terraform code won't continue/complete until all certs are issued.

This should work in a single apply run with no need to -target any resources in a first pass, though there is an apparently known issue around how long it takes for the validations to complete when performed via terraform, and for this reason it may require a second pass, albeit without changing the code or plan/apply invocation.

Answer 2

So after a bit of experimenting, I ended up leveraging -target=aws_acm_certificate.cert as a workaround to avoid the missing attribute errors I was seeing. The syntax I was using above was correct, and the error was a result of the apply needing to complete for the certificate before the validation steps could reference the generated attributes.

In addition, I found an elegant solution for the MANUAL FILL step using zipmap. The result looks like this...

Variable:

hosted_zones = [
  "foo.com"
]

Output from hosted_zones module:

output "hosted_zone_ids" {
  value = "${zipmap(var.hosted_zones, aws_route53_zone.zones.*.zone_id)}"
}

Then, my certificate generation/validation module looks like this, where var.hosted_zone_map is the output of the previous zipmap that creates a map of hosted zone domain name to assigned zone ID:

resource "aws_acm_certificate" "cert" {
    count                       = "${length(keys(var.hosted_zone_map))}"
    domain_name                 = "${element(keys(var.hosted_zone_map), count.index)}"
    subject_alternative_names   = ["*.${element(keys(var.hosted_zone_map), count.index)}"]
    validation_method           = "DNS"

    tags {
        Project     = "${var.project}"
        Environment = "${var.environment}"
    }
}

resource "aws_route53_record" "cert_validation" {
    count   = "${length(keys(var.hosted_zone_map))}"

    zone_id = "${lookup(var.hosted_zone_map, element(keys(var.hosted_zone_map), count.index))}"
    name = "${aws_acm_certificate.cert.*.domain_validation_options.0.resource_record_name[count.index]}"
    type = "${aws_acm_certificate.cert.*.domain_validation_options.0.resource_record_type[count.index]}"
    records = ["${aws_acm_certificate.cert.*.domain_validation_options.0.resource_record_value[count.index]}"]
    ttl     = 60
}

resource "aws_acm_certificate_validation" "cert" {
    count                   = "${length(keys(var.hosted_zone_map))}"

    certificate_arn         = "${aws_acm_certificate.cert.*.arn[count.index]}"
    validation_record_fqdns = ["${aws_route53_record.cert_validation.*.fqdn[count.index]}"]
}

The positioning of the splat was definitely the trickiest and least documented part of tracking this down, so hopefully this helps someone else out.