What does "eksctl create iamserviceaccount" do under the hood on an EKS cluster?

Question

AWS supports IAM Roles for Service Accounts (IRSA) that allows cluster operators to map AWS IAM Roles to Kubernetes Service Accounts.

To do so, one has to create an iamserviceaccount in an EKS cluster:

eksctl create iamserviceaccount \
    --name <AUTOSCALER_NAME> \
    --namespace kube-system \
    --cluster <CLUSTER_NAME> \
    --attach-policy-arn <POLICY_ARN> \
    --approve \
    --override-existing-serviceaccounts

The problem is that I don't want to use the above eksctl command because I want to declare my infrastructure using terraform.

Does eksctl command do anything other than creating a service account? If it only creates a service account, what is the YAML representation of it?

Answer 1

I am adding my answer here because I stumble upon the same issue, and accepted answer (and other answers above), do not provide full resolution to the issue - no code examples. They are just guidelines which I had to use to research much deeper. There are some issues which is really easy to miss - and without code examples its quite hard to conclude what is happening (especially part related with Conditions/StringEquals while creating IAM role)

The whole purpose of creating a service account which is going to be tied with the role - is possibility of creating aws resources from within cluster (most common case is load balancer, or roles for pushing logs to the cloudwatch).

So, question is how we can do this, using terraform, instead of using eks commands.

What we need to do, is:

1. create eks oidc (which can be done with terraform)
2. create AWS IAM role (which can be done with terraform), create and use proper policies
3. Create k8s service account (needs to be done with kubectl commands - or with terraform using kubernetes resources
4. Annotate k8s service account with IAM role we created (meaning that we are linking k8s service account with IAM role)

After this setup, our k8s service account will have k8s cluster role and k8s cluster role binding (which will allow that service account to perform actions within the k8s) and, our k8s service account will have IAM role attached to it, which will allow to perform actions outside of the cluster (like creating aws resources)

So lets start with it. Assumption bellow is that your eks cluster is already created with terraform, and we are focusing on creating resources areound that eks cluster necessary for working service account.

Create eks_oidc

### First we need to create tls certificate
data "tls_certificate" "eks-cluster-tls-certificate" {
  url = aws_eks_cluster.eks-cluster.identity[0].oidc[0].issuer
}

# After that create oidc
resource "aws_iam_openid_connect_provider" "eks-cluster-oidc" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.tls_certificate.eks-cluster-tls-certificate.certificates[0].sha1_fingerprint]
  url             = aws_eks_cluster.eks-cluster.identity[0].oidc[0].issuer
}

Now, lets create AWS IAM role with all necessary policies.

Terraform declarative code bellow will:

• create ALBIngressControllerIAMPolicy policy
• create alb-ingress-controller-role role
• attach ALBIngressControllerIAMPolicyr policy to alb-ingress-controller-role role
• attach already existing AmazonEKS_CNI_Policy policy to the role

Make a note that i used suffixes as alb ingress controller here, because that is primary use of my role from within the cluster. You can change the name of policy of the role or you can change permission access for the policy as well in dependency of what you are planing to do with it.

data "aws_caller_identity" "current" {}
locals {
  account_id = data.aws_caller_identity.current.account_id
  eks_oidc = replace(replace(aws_eks_cluster.eks-cluster.endpoint, "https://", ""), "/\\..*$/", "")
}

# Policy which will allow us to create application load balancer from inside of cluster
resource "aws_iam_policy" "ALBIngressControllerIAMPolicy" {
  name        = "ALBIngressControllerIAMPolicy"
  description = "Policy which will be used by role for service - for creating alb from within cluster by issuing declarative kube commands"

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect = "Allow",
        Action = [
          "elasticloadbalancing:ModifyListener",
          "wafv2:AssociateWebACL",
          "ec2:AuthorizeSecurityGroupIngress",
          "ec2:DescribeInstances",
          "wafv2:GetWebACLForResource",
          "elasticloadbalancing:RegisterTargets",
          "iam:ListServerCertificates",
          "wafv2:GetWebACL",
          "elasticloadbalancing:SetIpAddressType",
          "elasticloadbalancing:DeleteLoadBalancer",
          "elasticloadbalancing:SetWebAcl",
          "ec2:DescribeInternetGateways",
          "elasticloadbalancing:DescribeLoadBalancers",
          "waf-regional:GetWebACLForResource",
          "acm:GetCertificate",
          "shield:DescribeSubscription",
          "waf-regional:GetWebACL",
          "elasticloadbalancing:CreateRule",
          "ec2:DescribeAccountAttributes",
          "elasticloadbalancing:AddListenerCertificates",
          "elasticloadbalancing:ModifyTargetGroupAttributes",
          "waf:GetWebACL",
          "iam:GetServerCertificate",
          "wafv2:DisassociateWebACL",
          "shield:GetSubscriptionState",
          "ec2:CreateTags",
          "elasticloadbalancing:CreateTargetGroup",
          "ec2:ModifyNetworkInterfaceAttribute",
          "elasticloadbalancing:DeregisterTargets",
          "elasticloadbalancing:DescribeLoadBalancerAttributes",
          "ec2:RevokeSecurityGroupIngress",
          "elasticloadbalancing:DescribeTargetGroupAttributes",
          "shield:CreateProtection",
          "acm:DescribeCertificate",
          "elasticloadbalancing:ModifyRule",
          "elasticloadbalancing:AddTags",
          "elasticloadbalancing:DescribeRules",
          "ec2:DescribeSubnets",
          "elasticloadbalancing:ModifyLoadBalancerAttributes",
          "waf-regional:AssociateWebACL",
          "tag:GetResources",
          "ec2:DescribeAddresses",
          "ec2:DeleteTags",
          "shield:DescribeProtection",
          "shield:DeleteProtection",
          "elasticloadbalancing:RemoveListenerCertificates",
          "tag:TagResources",
          "elasticloadbalancing:RemoveTags",
          "elasticloadbalancing:CreateListener",
          "elasticloadbalancing:DescribeListeners",
          "ec2:DescribeNetworkInterfaces",
          "ec2:CreateSecurityGroup",
          "acm:ListCertificates",
          "elasticloadbalancing:DescribeListenerCertificates",
          "ec2:ModifyInstanceAttribute",
          "elasticloadbalancing:DeleteRule",
          "cognito-idp:DescribeUserPoolClient",
          "ec2:DescribeInstanceStatus",
          "elasticloadbalancing:DescribeSSLPolicies",
          "elasticloadbalancing:CreateLoadBalancer",
          "waf-regional:DisassociateWebACL",
          "elasticloadbalancing:DescribeTags",
          "ec2:DescribeTags",
          "elasticloadbalancing:*",
          "elasticloadbalancing:SetSubnets",
          "elasticloadbalancing:DeleteTargetGroup",
          "ec2:DescribeSecurityGroups",
          "iam:CreateServiceLinkedRole",
          "ec2:DescribeVpcs",
          "ec2:DeleteSecurityGroup",
          "elasticloadbalancing:DescribeTargetHealth",
          "elasticloadbalancing:SetSecurityGroups",
          "elasticloadbalancing:DescribeTargetGroups",
          "shield:ListProtections",
          "elasticloadbalancing:ModifyTargetGroup",
          "elasticloadbalancing:DeleteListener"
        ],
        Resource = "*"
      }
    ]
  })
}

# Create IAM role
resource "aws_iam_role" "alb-ingress-controller-role" {
  name = "alb-ingress-controller"

  assume_role_policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Federated": "${aws_iam_openid_connect_provider.eks-cluster-oidc.arn}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${replace(aws_iam_openid_connect_provider.eks-cluster-oidc.url, "https://", "")}:sub": "system:serviceaccount:kube-system:alb-ingress-controller",
          "${replace(aws_iam_openid_connect_provider.eks-cluster-oidc.url, "https://", "")}:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}
POLICY

  depends_on = [aws_iam_openid_connect_provider.eks-cluster-oidc]

  tags = {
    "ServiceAccountName" = "alb-ingress-controller"
    "ServiceAccountNameSpace" = "kube-system"
  }
}

# Attach policies to IAM role
resource "aws_iam_role_policy_attachment" "alb-ingress-controller-role-ALBIngressControllerIAMPolicy" {
  policy_arn = aws_iam_policy.ALBIngressControllerIAMPolicy.arn
  role       = aws_iam_role.alb-ingress-controller-role.name
  depends_on = [aws_iam_role.alb-ingress-controller-role]
}

resource "aws_iam_role_policy_attachment" "alb-ingress-controller-role-AmazonEKS_CNI_Policy" {
  role       = aws_iam_role.alb-ingress-controller-role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  depends_on = [aws_iam_role.alb-ingress-controller-role]
}

After executing terraform above, you have successfully created terraform part of the resources. Now we need to create a k8s service account and bind IAM role with that service account.

Creating cluster role, cluster role binding and service account

You can use

https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/master/docs/examples/rbac-role.yaml

directly (from the master branch), but having in mind that we need to annotate the iam arn, i have tendency to download this file, update it and store it as updated within my kubectl config files.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/name: alb-ingress-controller
  name: alb-ingress-controller
rules:
  - apiGroups:
      - ""
      - extensions
    resources:
      - configmaps
      - endpoints
      - events
      - ingresses
      - ingresses/status
      - services
      - pods/status
    verbs:
      - create
      - get
      - list
      - update
      - watch
      - patch
  - apiGroups:
      - ""
      - extensions
    resources:
      - nodes
      - pods
      - secrets
      - services
      - namespaces
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/name: alb-ingress-controller
  name: alb-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: alb-ingress-controller
subjects:
  - kind: ServiceAccount
    name: alb-ingress-controller
    namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/name: alb-ingress-controller
  name: alb-ingress-controller
  namespace: kube-system
  annotations:
    eks.amazonaws.com/role-arn: <ARN OF YOUR ROLE HERE>
...

At the bottom of this file, you will notice annotation where you will need to place your ANR role.

Double check

And that would be it. After that you have a k8s service account which is connected with iam role.

Check with:

kubectl get sa -n kube-system
kubectl describe sa alb-ingress-controller -n kube-system

And you should get output similar to this (annotations is the most important part, because it confirms the attachment of iam role):

Name:                alb-ingress-controller
Namespace:           kube-system
Labels:              app.kubernetes.io/managed-by=Helm
                     app.kubernetes.io/name=alb-ingress-controller
Annotations:         eks.amazonaws.com/role-arn: <YOUR ANR WILL BE HERE>
                     meta.helm.sh/release-name: testrelease
                     meta.helm.sh/release-namespace: default
Image pull secrets:  <none>
Mountable secrets:   alb-ingress-controller-token-l4pd8
Tokens:              alb-ingress-controller-token-l4pd8
Events:              <none>

From now on, you can use this service to manage internal k8s resources and external which are allowed by the policies you attached.

In my case, as mentioned before, I used it (beside other things) for creation of alb ingress controller and load balancer, hence all of the prefixes with "alb-ingress"