Kubernetes: kubectl top nodes/pods not working

Question

When I do kubectl top pods/nodes I am getting the following error:

metrics not available yet

When I check the logs I am getting

$ kubectl logs metrics-server-7df4c4484d-h75wr -n kube-system -c metrics-server

I1008 12:02:55.159069       1 serving.go:273] Generated self-signed cert(apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
[restful] 2018/10/08 12:02:56 log.go:33: [restful/swagger] listing is available at https://:443/swaggerapi
[restful] 2018/10/08 12:02:56 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I1008 12:02:56.358063       1 serve.go:96] Serving securely on [::]:443
E1008 12:03:04.225519       1 reststorage.go:101] unable to fetch node metrics for node "hpa-test": no metrics known for node "hpa-test"
E1008 12:03:07.619489       1 reststorage.go:101] unable to fetch node metrics for node "hpa-test": no metrics known for node "hpa-test"

Also, I am able to ping to hpa-test node from:

$ kubectl exec -it  metrics-server-7df4c4484d-h75wr -n kube-system sh

Also, I have tried looking for solution everywhere but nothing fixed the issue

Answer 1

There are two ways to fix this problem:

1) using heapster : installing heapster will allow 'kubectl top nodes' to work out of the box. However heapster has been deprecated, so you probably should switch to metrics-server.

2) using metrics-server : unfortunately it may not work out of the box when installing metrics-server... in case it doesn't, you need to update the end of the metrics-server-deployment.yaml (1.8+) file you used for installation and add a command section with the right parameters as follow:

containers:
- name: metrics-server
  image: k8s.gcr.io/metrics-server-amd64:v0.3.1
  imagePullPolicy: Always
  volumeMounts:
  - name: tmp-dir
    mountPath: /tmp

  command:
      - /metrics-server
      - --kubelet-insecure-tls
      - --kubelet-preferred-address-types=InternalIP

then simply apply the changes:

kubectl apply -f metrics-server-deployment.yaml

you should then be able to get results with

kubectl top nodes

and

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"

Answer 2

Need add flags for metrics-sever:

--kubelet-insecure-tls=true
--kubelet-port={YOU_KUBELET_PORT}
--kubelet-preferred-address-types=InternalIP
--v=5
--logtostderr