kubectl top node cannot work, it shows: "error: You must be logged in to the server (Unauthorized)" [closed]

Question

I tried to demploy a k8s environment for ver1.12_rc.1. It includes one master and 2 nodes. All of them are CentOS Linux release 7.4.1708 (Core). The related info are follows:

[root@bogon174 dashboard]# kubectl get pods -n kube-system
NAME                                     READY   STATUS    RESTARTS   AGE
heapster-684777c4cb-fm6kd                1/1     Running   0          32m
kubernetes-dashboard-77fd78f978-khc4f    1/1     Running   0          32m
metrics-server-v0.3.1-6879897646-c7rwz   2/2     Running   0          37m
monitoring-grafana-56b668bccf-29277      1/1     Running   0          32m
monitoring-influxdb-5c5bf4949d-l8ttc     1/1     Running   0          32m

[root@bogon174 dashboard]# kubectl get services -n kube-system
NAME                   TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)         AGE
heapster               ClusterIP   169.169.120.140   <none>        80/TCP          33m
kubernetes-dashboard   NodePort    169.169.151.109   <none>        443:26007/TCP   33m
metrics-server         NodePort    169.169.218.252   <none>        443:10521/TCP   38m
monitoring-grafana     ClusterIP   169.169.170.53    <none>        80/TCP          33m
monitoring-influxdb    ClusterIP   169.169.248.0     <none>        8086/TCP        33m

[root@bogon174 dashboard]# kubectl get nodes
NAME             STATUS     ROLES    AGE   VERSION
192.168.20.171   Ready      <none>   10d   v1.12.0-rc.1
192.168.20.172   NotReady   <none>   10d   v1.12.0-rc.1

I researched previous questions, but I cannot to get the correct solution.

Answer 1

For my friends on EKS make sure you have the username set (and not set to just the session name like I did) in the aws-auth configmap:

When I saw this:

robert ❱ kubectl top node
error: You must be logged in to the server (Unauthorized)
robert ❱ 1 ❱ kubectl logs -n kube-system -l app.kubernetes.io/instance=metrics-server
E0407 22:34:45.879156       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:34:49.399854       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:34:50.691133       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:34:51.827629       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:39:07.288163       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:39:08.755492       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:39:09.801957       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:40:32.405458       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=801591513699736721, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:43:09.791769       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
E0407 22:44:14.244221       1 authentication.go:63] "Unable to authenticate the request" err="verifying certificate SN=3949940469908359789, SKID=, AKID= failed: x509: certificate signed by unknown authority"
robert ❱

Here was my remediation.

1. Check the role configuration in the aws-auth config map:

robert ❱ kubectl get configmaps -n kube-system aws-auth -o yaml | grep MyTeamRole$ -A 3
- rolearn: arn:aws:iam::123456789012:role/MyTeamRole
  username: {{SessionName}}
  groups:
    - system:masters
robert ❱

NOTE: I had the username set to {{SessionName}} without a prefix.

2. Edit the config map like so:

robert ❱ kubectl edit configmap -n kube-system aws-auth
configmap/aws-auth edited
robert ❱ kubectl get configmaps -n kube-system aws-auth -o yaml | grep MyTeamRole$ -A 3
    - rolearn: arn:aws:iam::123456789012:role/MyTeamRole
      username: literally_anything:{{SessionName}}
      groups:
        - system:masters
robert ❱ 

3. Retry the kubectl top command (or whatever command you were running) with the new aws-auth configmap edits:

robert ❱ kubectl top node
NAME                                       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
ip-10-0-3-103.us-west-2.compute.internal   341m         17%    1738Mi          52%
...
robert ❱

Where:

• MyTeamRole is the name of the IAM role your team uses
• 123456789012 is your AWS Account ID
• literally_anything can be any string like my_team or iam