use different bucket for terraform s3 backend depending on which aws account is configured

Question

I need my terraform s3 backend to use one bucket for my production AWS account, and another bucket for my development AWS account. We need this because we can't allow users from the development AWS account to access the s3 bucket in the production AWS account. S3 bucket names must be globally unique, so the bucket name field can't be the same.

I tried using a variable here but variables error in terraform backends. This is a very requested feature, and there's a GitHub issue for it, but it's been open for 4 years and there doesn't seem to be any plan to add this feature, so I need a workaround. One suggestion comes from that same GitHub issue. That suggestion is to use a terraform remote state data source. Unfortunately, this doesn't seem to work. Here's what I tried:

// backend.tf

terraform {
  backend "s3" {}
}
data terraform_remote_state "state" {
  backend = "s3"
  config {
    bucket = var.aws_account == "123456789000" ? "my-prod-bucket" : "my-dev-bucket"
    key    = "apps/main-stack.tfstate"
    region = "us-east-1"
  }
}

None of the values get used, and it prompts for all of them to be entered manually.

$ terraform init
Initializing modules...

Initializing the backend...
bucket
The name of the S3 bucket

Enter a value:

I looked around for other solutions, but so far, no luck. Does anyone know how to fix this?

Answer 1

Seems like the solution is to use a backend-config file. This is called partial configuration. You can keep a tfvars file for each aws account you need a separate backend for, and provide it when you initialize terraform like so:

// prod-backend-config.tfvars

bucket = "my-prod-s3-bucket-for-terraform"

// dev-backend-config.tfvars

bucket = "my-dev-s3-bucket-for-terraform"

// backend.tf

terraform {
  backend "s3" {
    // do not set a bucket name here
    key    = "apps/main-stack.tfstate"
    region = "us-east-1"
  }
}

$ terraform init -backend-config prod-backend-config.tfvars

This allows you to maintain parity between aws accounts and s3 buckets for your backend.

For a more general solution to configuring any backend with arbitrary variables, see How to pass variables for Terraform S3 Backend resource?