Kubernetes Dashboard - Unknown server error after login

Question

I´ve successfully deployed Kubernetes via Kubespray and everything seems to work fine. I am able to access the cluster via kubectl and list nodes, pods, services, secrets and so on. It`s also possible to apply new ressources and dashboard endpoint gets me the dashboard login page.

I´ve logged in with tokens of different serviceaccounts (default, kubernetes-dashboard, kubernetes-admin, ...)...with every login I get the same popups as described in kubespray dashboard warning forbidden popups for example.

So I applied the clusterrolebinding for default service account as described. When I login now with the defaults account token, I only get a

Unknown Server Error (404)
the server could not find the requested resource
Redirecting to previous state in 3 seconds...

box which redirects me to the login page afterwards. Its the same behaviour if I connect to Dashboard via kubectl proxy. The access is HTTPS over a public cluster IP and also HTTP over proxy

I am using Kubernetes 1.16.2 and latest Kubespray master commit 18d19d9e

EDIT: I destroyed and reprovisioned the cluster to get a fresh Kubespray-provisioned instance to make all steps deterministic, adding more info...

kubectl -n kube-system logs --follow kubernetes-dashboard-556b9ff8f8-jbmgg -- during an login attempt gives me

2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Incoming HTTP/2.0 GET /api/v1/csrftoken/login request from 10.233.74.0:57458: { contents hidden }
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Incoming HTTP/2.0 POST /api/v1/login request from 10.233.74.0:57458: { contents hidden }
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Incoming HTTP/2.0 GET /api/v1/login/status request from 10.233.74.0:57458: {}
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Incoming HTTP/2.0 GET /api/v1/csrftoken/token request from 10.233.74.0:57458: {}
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Incoming HTTP/2.0 POST /api/v1/token/refresh request from 10.233.74.0:57458: { contents hidden }
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Incoming HTTP/2.0 GET /api/v1/login/status request from 10.233.74.0:57458: {}
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Incoming HTTP/2.0 GET /api/v1/csrftoken/token request from 10.233.74.0:57458: {}
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Incoming HTTP/2.0 POST /api/v1/token/refresh request from 10.233.74.0:57458: { contents hidden }
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:02 [2019-12-16T12:35:02Z] Incoming HTTP/2.0 GET /api/v1/overview/default?filterBy=&itemsPerPage=10&name=&page=1&sortBy=d,creationTimestamp request from 10.233.74.0:57458: {}
2019/12/16 12:35:03 Getting config category
2019/12/16 12:35:03 Getting discovery and load balancing category
2019/12/16 12:35:03 Getting lists of all workloads
2019/12/16 12:35:03 the server could not find the requested resource
2019/12/16 12:35:03 [2019-12-16T12:35:03Z] Outcoming response to 10.233.74.0:57458 with 404 status code
2019/12/16 12:35:03 No metric client provided. Skipping metrics.
2019/12/16 12:35:03 No metric client provided. Skipping metrics.
2019/12/16 12:35:03 No metric client provided. Skipping metrics.
2019/12/16 12:35:03 Getting pod metrics
2019/12/16 12:35:03 No metric client provided. Skipping metrics.
2019/12/16 12:35:03 No metric client provided. Skipping metrics.
2019/12/16 12:35:03 [2019-12-16T12:35:03Z] Incoming HTTP/2.0 GET /api/v1/systembanner request from 10.233.74.0:57458: {}
2019/12/16 12:35:03 [2019-12-16T12:35:03Z] Incoming HTTP/2.0 GET /api/v1/login/status request from 10.233.74.0:57458: {}
2019/12/16 12:35:03 [2019-12-16T12:35:03Z] Incoming HTTP/2.0 GET /api/v1/rbac/status request from 10.233.74.0:57458: {}
2019/12/16 12:35:03 [2019-12-16T12:35:03Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:03 [2019-12-16T12:35:03Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:03 [2019-12-16T12:35:03Z] Outcoming response to 10.233.74.0:57458 with 200 status code
2019/12/16 12:35:12 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2019/12/16 12:35:42 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.

I`ve found a strange workaround to get dashboard working, but this is not usable for us in production, perhaps somebody can explain this:

1. I take for example the serviceaccount kube-system:default (Note: this one is NOT assigned cluster-admin at this point
2. I get its token and login with that
3. Dashboard obviously shows me the "forbidden-popups"
4. While still logged in, I run kubectl create clusterrolebinding default-admin --clusterrole cluster-admin --serviceaccount=kube-system:default
5. I refresh the browser tab which holds my dashboard session...et voila, everything shows up correctly.

Therefore I am not able to logout and login again, I always have to delete the clusterrolebinding, login then and afterwards apply clusterrolebinding again.

This seems to be strongly related to kubespray provisioned clusters, so anybody able to reproduce this with kubespray?

Answer 1

If your are using certificate to connect you certificate should be in the system:masters group So include the "Subject: O=system:masters, CN="

You can also create a Token and then use the token instead of the certificate:

There might be a possibility that your cluster role is bound to "Service Account" but not your group, You should check your group in the yaml file.Your service account has an access token, use that to authenticate instead of your certificate.

Use this to create a token and use it.

kubectl describe secret $(kubectl get secret | grep cluster-admin | awk '{print $1}')

token:

Update kubeconfig to authenticate yourself using that token, instead of the certificate you are currently using, and you should be successfully authenticated as that cluster-admin service account.

Kubernetes RBAC - forbidden attempt to grant extra privileges

Answer 2

Alright, this seems to be a bug which is issued in Kubespray Github repo issue #5347