eks iam roles for services account not working

Question

I'm trying my hand on iam roles for services account to secure the autoscaller. But I seem to be missing something. Little precision I'm using terraform to create the cluster.

I followed these documentation:

• https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/
• https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

So I've created a role other than the one for the nodes and applied the policy for the autoscaller to this new role. This part is basic, no issue there.

I also activated the openid provider in terraform:

resource "aws_iam_openid_connect_provider" "example" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = []
  url             = aws_eks_cluster.eks.identity.0.oidc.0.issuer
}

No issue the cluster is creating itself with no issue.

No I added the annotation to service account for the autoscalling:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::ID:role/terraform-eks-autoscaller
  labels:
    k8s-addon: cluster-autoscaler.addons.k8s.io
    k8s-app: cluster-autoscaler
  name: cluster-autoscaler
  namespace: kube-system

My problem is that it does not seems to works and the pod is still trying to use the new IAM role but still using the node role:

Failed to create AWS Manager: cannot autodiscover ASGs: AccessDenied: User: arn:aws:sts::ID:assumed-role/terraform-eks-node/i-ID is not authorized to perform: autoscaling:DescribeTags

Does someone know what step I'm missing here?

Thanks in advance for the help ;)

Answer 1

So answer is very simple. Your OIDC provider configuration is missing the thumbprint. It is essential for Iam to work correctly. Normally if you create OIDC provider in AWS console that thumbprint gets populated automatically, however it is not the case when you do it through terraform.

I have been caught by this as well so I have written a blog about this that you can find here: https://medium.com/@marcincuber/amazon-eks-with-oidc-provider-iam-roles-for-kubernetes-services-accounts-59015d15cb0c

To solve your issue simply add the following:

9E99A48A9960B14926BB7F3B02E22DA2B0AB7280

The above is the hashed root CA that doesn’t change for another 10+ years and it is the same across all regions. How to acquire it, you can read the blog I added link to above.

Additionally, ensure to use the latest autoscaler version which is matching the version of your kubernetes. Also, try adding security context with fsGroup: 65534. That is the current workaround to make the OIDC work properly for some apps.