Terraform: Error: Kubernetes cluster unreachable: invalid configuration

Question

After deleting kubernetes cluster with "terraform destroy" I can't create it again anymore.

"terraform apply" returns the following error message:

Error: Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

Here is the terraform configuration:

terraform {
  backend "s3" {
    bucket = "skyglass-msur"
    key    = "terraform/backend"
    region = "us-east-1"
  }
}

locals {
  env_name         = "staging"
  aws_region       = "us-east-1"
  k8s_cluster_name = "ms-cluster"
}

variable "mysql_password" {
  type        = string
  description = "Expected to be retrieved from environment variable TF_VAR_mysql_password"
}

provider "aws" {
  region = local.aws_region
}

data "aws_eks_cluster" "msur" {
  name = module.aws-kubernetes-cluster.eks_cluster_id
}

module "aws-network" {
  source = "github.com/skyglass-microservices/module-aws-network"

  env_name              = local.env_name
  vpc_name              = "msur-VPC"
  cluster_name          = local.k8s_cluster_name
  aws_region            = local.aws_region
  main_vpc_cidr         = "10.10.0.0/16"
  public_subnet_a_cidr  = "10.10.0.0/18"
  public_subnet_b_cidr  = "10.10.64.0/18"
  private_subnet_a_cidr = "10.10.128.0/18"
  private_subnet_b_cidr = "10.10.192.0/18"
}

module "aws-kubernetes-cluster" {
  source = "github.com/skyglass-microservices/module-aws-kubernetes"

  ms_namespace       = "microservices"
  env_name           = local.env_name
  aws_region         = local.aws_region
  cluster_name       = local.k8s_cluster_name
  vpc_id             = module.aws-network.vpc_id
  cluster_subnet_ids = module.aws-network.subnet_ids

  nodegroup_subnet_ids     = module.aws-network.private_subnet_ids
  nodegroup_disk_size      = "20"
  nodegroup_instance_types = ["t3.medium"]
  nodegroup_desired_size   = 1
  nodegroup_min_size       = 1
  nodegroup_max_size       = 5
}

# Create namespace
# Use kubernetes provider to work with the kubernetes cluster API
provider "kubernetes" {
  # load_config_file       = false
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.msur.certificate_authority.0.data)
  host                   = data.aws_eks_cluster.msur.endpoint
  exec {
    api_version = "client.authentication.k8s.io/v1alpha1"
    command     = "aws-iam-authenticator"
    args        = ["token", "-i", "${data.aws_eks_cluster.msur.name}"]
  }
}

# Create a namespace for microservice pods
resource "kubernetes_namespace" "ms-namespace" {
  metadata {
    name = "microservices"
  }
}

P.S. There seems to be the issue with terraform kubernetes provider for 0.14.7

I couldn't use "load_config_file" = false in this version, so I had to comment it, which seems to be the reason of this issue.

P.P.S. It could also be the issue with outdated cluster_ca_certificate, which terraform tries to use: deleting this certificate could be enough, although I'm not sure, where it is stored.

Answer 1

Before doing something radical like manipulating the state directly, try setting the KUBE_CONFIG_PATH variable:

export KUBE_CONFIG_PATH=/path/to/.kube/config

After this rerun the plan or apply command. This has fixed the issue for me.

Answer 2

I had the same issue. I even manually deleted the EKS cluster which really messed up the terraform state.

However, after wasting a few hours, I found out that there is a very simple solution.

You can run

terraform state rm <resource_type>.<resource_name>

I just executed

terraform state rm `terraform state list | grep eks`

to remove all the entries for a particular service from state file in a safe manner.