AWS-CNI fails with failed to connect service ":50051" [closed]

Question

I'm trying to setup an unmanaged k8s cluster on raw AWS EC2 instances. I've installed the control-plane which runs fine. Only core-dns is in a perpetual state of unknown, which is to be expected, until a CNI is installed.

ubuntu@ip-xxx-xx-1-11:/var/log/aws-routed-eni$ kubectl get po -n kube-system
NAME                                     READY   STATUS             RESTARTS      AGE
aws-node-hjx46                           0/1     CrashLoopBackOff   9 (0s ago)    24m
coredns-64897985d-2z7xd                  0/1     Unknown            1             81m
coredns-64897985d-7tplp                  0/1     Unknown            1             81m
etcd-ip-xxx-xx-1-11                      1/1     Running            3 (25m ago)   81m
kube-apiserver-ip-xxx-xx-1-11            1/1     Running            3 (25m ago)   81m
kube-controller-manager-ip-xxx-xx-1-11   1/1     Running            3 (25m ago)   81m
kube-proxy-ktbfd                         1/1     Running            3 (25m ago)   81m
kube-scheduler-ip-xxx-xx-1-11            1/1     Running            3 (25m ago)   81m

(The 3 restarts are due to me rebooting the node)

I've added an IAM role with the following policy to my control-plane node:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:AssignPrivateIpAddresses",
                "ec2:AttachNetworkInterface",
                "ec2:CreateNetworkInterface",
                "ec2:DeleteNetworkInterface",
                "ec2:DescribeInstances",
                "ec2:DescribeTags",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeInstanceTypes",
                "ec2:DetachNetworkInterface",
                "ec2:ModifyNetworkInterfaceAttribute",
                "ec2:UnassignPrivateIpAddresses"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags"
            ],
            "Resource": [
                "arn:aws:ec2:*:*:network-interface/*"
            ]
        }
    ]
}

Also I have create a security-group which allows all traffic on all ports from any node with the same security-group assigned. So my control-plane node should be able to communicate to it-self on any port.

I then downloaded aws-k8s-cni.yaml v1.11.4 from github, and install it using:

kubectl apply -f aws-k8s-cni.yaml

After installation I get: "msg":"Retrying waiting for IPAM-D" in the log, and "timeout: failed to connect service \":50051\" within 5s" when I describe my aws-node pod.

However doing a cat /var/log/aws-routed-eni/ipamd.log | grep "error" yields no results whatsoever. Which, after perusing the file, leads me to believe, that ipamd runs without a hitch.

os: Ubuntu 22.04
k8s: 1.23.12
aws-cni: 1.11.4

EC2 instance has been tagged with kubernetes.io/cluster/<cluster-name>: owned

and kubelet starts with --cloud-provider=aws

Any help will be greatly appreciated!

Thank you!

Answer 1

If using VPC CNI with IPv6, you may have transitioned from a dual IPv4 CIDR model (that required multiple sets of subnets for pods vs hosts), ensure you have Custom networking disabled. This carried over from our previous environment.

I looked at /var/log/aws-routed-eni/ipamd.log and saw the following.

{"level":"error","ts":"2023-11-30T23:24:53.119Z","caller":"ipamd/ipamd.go:433","msg":"IPv6 is supported only in Prefix Delegation mode. Security Group Per Pod and Custom Networking are not supported in IPv6 mode. Please set the env variables accordingly."}

To fix this, I ran kubectl set env daemonset aws-node -n kube-system AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=false

Sources:

• VPC CNI Custom Networking documentation

• VPC CNI Assign IPv6 addresses to pods