Layered deployments with Terraform

Question

I am new to Terraform so not even sure something like this is possible. As an example, lets say I have a template that deploys an Azure resource group and a key vault in it. And then lets say I have another template that deploys a virtual machine into the same resource group. Is it possible to do a destroy with the virtual machine template without destroying the key vault and resource group? We are trying to compartmentalize the parts of a large solution without having to put it all in a single template and we want to be able to manage each piece separately without affecting other pieces.

On a related note...we are storing state files in an Azure storage account. If we breakup our deployment into multiple compartmentalized deployments...should each deployment have its own state file or should they all use the same state file?

Answer 1

For larger systems it is common to split infrastructure across multiple separate configurations and apply each of them separately. This is a separate idea from (and complimentary to) using shared modules: modules allow a number of different configurations to have their own separate "copy" of a particular set of infrastructure, while the patterns described below allow an object managed by one configuration to be passed by reference to another.

If some configurations will depend on the results of other configurations, it's necessary to store these results in some data store that can be written to by its producer and read from by its consumer. In an environment where the Terraform state is stored remotely and readable broadly, the terraform_remote_state data source is a common way to get started:

data "terraform_remote_state" "resource_group" {
  # The settings here should match the "backend" settings in the
  # configuration that manages the network resources.
  backend = "s3"
  config {
    bucket = "mycompany-terraform-states"
    region = "us-east-1"
    key    = "azure-resource-group/terraform.tfstate"
  }
}

resource "azurerm_virtual_machine" "example" {
  resource_group_name = "${data.terraform_remote_state.resource_group.resource_group_name}"
  # ... etc ...
}

The resource_group_name attribute exported by the terraform_remote_state data source in this example assumes that a value of that name was exposed by the configuration that manages the resource group using an output.

This decouples the two configurations so that they have an entirely separate lifecycle. You first terraform apply in the configuration that creates the resource group, and then terraform apply in the configuration that contains the terraform_remote_state data resource shown above. You can then apply that latter configuration as many times as you like without risk to the shared resource group or key vault.

---

While the terraform_remote_state data source is quick to get started with for any organization already using remote state (which is recommended), some organizations prefer to decouple configurations further by introducing an intermediate data store like Consul, which then allows data to be passed between configurations more explicitly.

To do this, the "producing" configuration (the one that manages your resource group) publishes the necessary information about what it created into Consul at a well-known location, using the consul_key_prefix resource:

resource "consul_key_prefix" "example" {
  path_prefix = "shared/resource_group/"
  subkeys = {
    name = "${azurerm_resource_group.example.name}"
    id   = "${azurerm_resource_group.example.id}"
  }

resource "consul_key_prefix" "example" {
  path_prefix = "shared/key_vault/"
  subkeys = {
    name = "${azurerm_key_vault.example.name}"
    id   = "${azurerm_key_vault.example.id}"
    uri  = "${azurerm_key_vault.example.uri}"
  }
}

The separate configuration(s) that use the centrally-managed resource group and key vault would then read it using the consul_keys data source:

data "consul_keys" "example" {
  key {
    name = "resource_group_name"
    path = "shared/resource_group/name"
  }
  key {
    name = "key_vault_name"
    path = "shared/key_vault/name"
  }
  key {
    name = "key_vault_uri"
    path = "shared/key_vault/uri"
  }
}

resource "azurerm_virtual_machine" "example" {
  resource_group_name = "${data.consul_keys.example.var.resource_group_name}"
  # ... etc ...
}

In return for the additional complexity of running another service to store these intermediate values, the two configurations now know nothing about each other apart from the agreed-upon naming scheme for keys within Consul, which gives flexibility if, for example, in future you decide to refactor these Terraform configurations so that the key vault has its own separate configuration too. Using a generic data store like Consul also potentially makes this data available to the applications themselves, e.g. via consul-template.

Consul is just one example of a data store that happens to already be well-supported in Terraform. It's also possible to achieve similar results using any other data store that Terraform can both read and write. For example, you could even store values in TXT records in a DNS zone and use the DNS provider to read, as an "outside the box" solution that avoids running an additional service.

---

As usual, there is a tradeoff to be made here between simplicity (with "everything in one configuration" being the simplest possible) and flexibility (with a separate configuration store), so you'll need to evaluate which of these approaches is the best fit for your situation.

As some additional context: I've documented a pattern I used successfully for a moderate-complexity system. In that case we used a mixture of Consul and DNS to create an "environment" abstraction that allowed us to deploy the same applications separately for a staging environment, production, etc. The exact technologies used are less important than the pattern, though. That approach won't apply exactly to all other situations, but hopefully there are some ideas in there to help others think about how to best make use of Terraform in their environment.