"kubectl exec" results in "error: unable to upgrade connection: Unauthorized"

Question

I tried kubectl exec on a k8s 1.6.4 RBAC-enabled cluster and the error returned was: error: unable to upgrade connection: Unauthorized. docker exec on the same container succeeds. Otherwise, kubectl is working. kubectl tunnels through an SSH connection but I don't think this is the issue.

kubelet authn is enabled but not authz. The docs say that authz is AlwaysAllow by default, so I have left it this way.

I have a feeling that it is similar to this issue. But the error message is a tad different.

Thanks in advance!

Verbose logs for the kubectl exec command:

I0614 16:50:11.003677   64104 round_trippers.go:398] curl -k -v -XPOST  -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" https://localhost:6443/api/v1/namespaces/monitoring/pods/alertmanager-main-0/exec?command=%2Fbin%2Fls&container=alertmanager&container=alertmanager&stderr=true&stdout=true
I0614 16:50:11.003705   64104 round_trippers.go:398] curl -k -v -XPOST  -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "User-Agent: kubectl/v1.6.4 (darwin/amd64) kubernetes/d6f4332" https://localhost:6443/api/v1/namespaces/monitoring/pods/alertmanager-main-0/exec?command=%2Fbin%2Fls&container=alertmanager&container=alertmanager&stderr=true&stdout=true
I0614 16:50:11.169474   64104 round_trippers.go:417] POST https://localhost:6443/api/v1/namespaces/monitoring/pods/alertmanager-main-0/exec?command=%2Fbin%2Fls&container=alertmanager&container=alertmanager&stderr=true&stdout=true 401 Unauthorized in 165 milliseconds
I0614 16:50:11.169493   64104 round_trippers.go:423] Response Headers:
I0614 16:50:11.169497   64104 round_trippers.go:426]     Date: Wed, 14 Jun 2017 08:50:11 GMT
I0614 16:50:11.169500   64104 round_trippers.go:426]     Content-Length: 12
I0614 16:50:11.169502   64104 round_trippers.go:426]     Content-Type: text/plain; charset=utf-8
I0614 16:50:11.169506   64104 round_trippers.go:417] POST https://localhost:6443/api/v1/namespaces/monitoring/pods/alertmanager-main-0/exec?command=%2Fbin%2Fls&container=alertmanager&container=alertmanager&stderr=true&stdout=true 401 Unauthorized in 165 milliseconds
I0614 16:50:11.169509   64104 round_trippers.go:423] Response Headers:
I0614 16:50:11.169512   64104 round_trippers.go:426]     Date: Wed, 14 Jun 2017 08:50:11 GMT
I0614 16:50:11.169545   64104 round_trippers.go:426]     Content-Length: 12
I0614 16:50:11.169548   64104 round_trippers.go:426]     Content-Type: text/plain; charset=utf-8
F0614 16:50:11.169635   64104 helpers.go:119] error: unable to upgrade connection: Unauthorized

Answer 1

This is an RTFM moment... The solution was basically to follow all the steps on this page for authn, authz, or both.

I had omitted --kubelet-client-certificate and --kubelet-client-key which resulted in the error. Without these flags, kube-apiserver will fail to authenticate with kubelet when you do a kubectl exec.

My original attempt to configure authn was by reading the docs for the kubelet daemon (ie. not the one above). Hence the grave omission.

Answer 2

In my case (while learning Kubernetes The Hard Way, I had to configure RBAC permissions to allow the Kubernetes API Server to access the Kubelet API on each worker node. I had created a ClusterRole and ClusterRoleBinding to access the Kubelet API

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
    verbs:
      - "*"

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kube-apiserver

References:

• https://github.com/mmumshad/kubernetes-the-hard-way/blob/master/docs/13-kube-apiserver-to-kubelet.md