Terraform EKS tagging

Question

I am having this issue of Terraform EKS tagging and don't seem to find workable solution to tag all the VPC subnets when a new cluster is created.

To provide some context: We have one AWS VPC where we deployment several EKS cluster into the subnets. We do not create VPC or subnets are part of the EKS cluster creation. Therefore, the terraform code creating a cluster doesn't get to tag existing subnets and VPC. Although EKS will add the required tags, they are automatically removed next time we run terraform apply on the VPC.

My attempt to workaround is to provide a terraform.tfvars file within the VPC to as follows:

eks_tags = 
 [
 "kubernetes.io/cluster/${var.cluster-1}", "shared", 
 "kubernetes.io/cluster/${var.cluster-2}", "shared",
 "kubernetes.io/cluster/${var.cluster-2}", "shared",
]    

Then within the VPC and subnets resources, we do something like

    resource "aws_vpc" "demo" {
      cidr_block = "10.0.0.0/16"

      tags = "${
        map(
         ${var.eks_tags}
        )
     }"
    }

However, the above does not seem to work. I have tried various Terraform 0.11 functions from https://www.terraform.io/docs/configuration-0-11/interpolation.html but not of them help.

Has anyone ben able to resolve this issue?

The idea that we always create new VPC and subnet for every EKS cluster is wrong. Obviously, the has to be a way to tag existing VPC and subnet resources using Terraform?

Answer 1

You can now use the aws provider ignore_tags attribute so that the tags made with the aws_ec2_tag resource do not get removed next time the VPC module is applied.

For example the provider becomes:

provider "aws" {
  profile = "terraform"
  region  = "us-west-1"
  
  // This is necessary so that tags required for eks can be applied to the vpc without changes to the vpc wiping them out.
  // https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/resource-tagging
  ignore_tags {
    key_prefixes = ["kubernetes.io/"]
  }
}

You can then leverage the aws_ec2_tag resource like so in your EKS module without worrying about the tag getting removed next time the VPC module is applied.

/**
  Start of resource tagging logic to update the provided vpc and its subnets with the necessary tags for eks to work
  The toset() function is actually multiplexing the resource block, one for every item in the set. It is what allows 
  for setting a tag on each of the subnets in the vpc.
*/
resource "aws_ec2_tag" "vpc_tag" {
  resource_id = data.terraform_remote_state.vpc.outputs.vpc_id
  key         = "kubernetes.io/cluster/${var.cluster_name}"
  value       = "shared"
}

resource "aws_ec2_tag" "private_subnet_cluster_tag" {
  for_each    = toset(data.terraform_remote_state.vpc.outputs.private_subnets)
  resource_id = each.value
  key         = "kubernetes.io/cluster/${var.cluster_name}"
  value       = "shared"
}

resource "aws_ec2_tag" "public_subnet_cluster_tag" {
  for_each    = toset(data.terraform_remote_state.vpc.outputs.public_subnets)
  resource_id = each.value
  key         = "kubernetes.io/cluster/${var.cluster_name}"
  value       = "shared"
}

/**
  These tags have been decoupled from the eks module and moved to the more appropirate vpc module.
*/
resource "aws_ec2_tag" "private_subnet_tag" {
  for_each    = toset(data.terraform_remote_state.vpc.outputs.private_subnets)
  resource_id = each.value
  key         = "kubernetes.io/role/internal-elb"
  value       = "1"
}

resource "aws_ec2_tag" "public_subnet_tag" {
  for_each    = toset(data.terraform_remote_state.vpc.outputs.public_subnets)
  resource_id = each.value
  key         = "kubernetes.io/role/elb"
  value       = "1"
}

Answer 2

This problem will always exist when there are 2 pieces of code with different state files trying to act on the same resource.

One way to solve this is to re-import the VPC resource into your VPC state file everytime you apply your EKS terraform code. This will import your tags as well. Same goes with subnets, but it is a manual and tedious process in the long run.

terraform import aws_vpc.test_vpc vpc-a01106c2

Ref: https://www.terraform.io/docs/providers/aws/r/vpc.html

Cheers!

Answer 3

In our case we have separate scripts to provision VPC and networking resources there we are not adding EKS specific tags.

For EKS cluster provisioning we have separate scripts which will auto update/add tags on cluster.

So on VPC scripts in provider.tf file we add below condition so that scripts will not remove these tags and everything works properly.

provider "aws" {
region = "us-east-1"
 ignore_tags {
    key_prefixes = ["kubernetes.io/cluster/"]
  }
}