Effective GitLab CI/CD workflow with multiple Terraform configurations?

Question

My team uses AWS for our infrastructure, across 3 different AWS accounts. We'll call them simply sandbox, staging, and production.

I recently set up Terraform against our AWS infrastructure, and its hierarchy maps against our accounts, then by either application, or AWS service itself. The repo structure looks something like this:

staging
  iam
    groups
      main.tf
    users
      main.tf
  s3
    main.tf
sandbox
  iam
    ...
production
  applications
    gitlab
      main.tf
  route53
    main.tf
  ...

We're using separate configurations per AWS service (e.g., IAM or S3) or application (e.g., GitLab) so we don't end up with huge .tf files per account that would take a long time to apply updates for any one change. Ideally, we'd like to move away from the service-based configuration approach and move towards more application-based configurations, but the problem at hand remains the same either way.

This approach has been working fine when applying updates manually from the command line, but I'd love to move it to GitLab CI/CD to better automate our workflow, and that's where things have broken down.

In my existing setup, if I make an single change to, say, staging/s3/main.tf, GitLab doesn't seem to have a good way out of the box to only run terraform plan or terraform apply for that specific configuration.

If I instead moved everything into a single main.tf file for an entire AWS account (or multiple files but tied to a single state file), I could simply have GitLab trigger a job to do plan or apply to just that configuration. It might take 15 minutes to run based on the number of AWS resources we have in each account, but it's a potential option I suppose.

It seems like my issue might be ultimately related to how GitLab handles "monorepos" than how Terraform handles its workflow (after all, Terraform will happily plan/apply my changes if I simply tell it what has changed), although I'd also be interested in hearing about how people structure their Terraform environments given -- or in order to avoid entirely -- these limitations.

Has anyone solved an issue like this in their environment?

Answer 1

The nice thing about Terraform is that it's idempotent so you can just apply even if nothing has changed and it will be a no-op action anyway.

If for some reason you really only want to run a plan/apply on a specific directory when things change then you can achieve this by using only.changes so that Gitlab will only run the job if the specified files have changed.

So if you have your existing structure then it's as simple as doing something like this:

stages:
  - terraform plan
  - terraform apply

.terraform_template:
  image: hashicorp/terraform:latest
  before_script:
    - LOCATION=$(echo ${CI_JOB_NAME} | cut -d":" -f2)
    - cd ${LOCATION}
    - terraform init

.terraform_plan_template:
  stage: terraform plan
  extends: .terraform_template
  script:
    - terraform plan -input=false -refresh=true -module-depth=-1 .

.terraform_apply_template:
  stage: terraform apply
  extends: .terraform_template
  script:
    - terraform apply -input=false -refresh=true -auto-approve=true .

terraform-plan:production/applications/gitlab:
  extends: .terraform_plan_template
  only:
    refs:
      - master
    changes:
      - production/applications/gitlab/*
      - modules/gitlab/*

terraform-apply:production/applications/gitlab:
  extends: .terraform_apply_template
  only:
    refs:
      - master
    changes:
      - production/applications/gitlab/*
      - modules/gitlab/*

I've also assumed the existence of modules that are in a shared location to indicate how this pattern can also look for changes elsewhere in the repo than just the directory you are running Terraform against.

If this isn't the case and you have a flatter structure and you're happy to have a single apply job then you can simplify this to something like:

stages:
  - terraform

.terraform_template:
  image: hashicorp/terraform:latest
  stage: terraform
  before_script:
    - LOCATION=$(echo ${CI_JOB_NAME} | cut -d":" -f2)
    - cd ${LOCATION}
    - terraform init
  script:
    - terraform apply -input=false -refresh=true -auto-approve=true .
  only:
    refs:
      - master
    changes:
      - ${CI_JOB_NAME}/*

production/applications/gitlab:
  extends: .terraform_template

In general though this can just be avoided by allowing Terraform to run against all of the appropriate directories on every push (probably only applying on push to master or other appropriate branch) because, as mentioned, Terraform is idempotent so it won't do anything if nothing has changed. This also has the benefit that if your automation code hasn't changed but something has changed in your provider (such as someone opening up a security group) then Terraform will go put it back to how it should be the next time it is triggered.