kubectl unauthorized when using service account token inside gitlab-runner environment

Question

We're running gitlab-runner instances via kubernetes executors inside a kubernetes cluster (let's call it KUBE01). These instances build and deploy to the kubernetes cluster, and runners are given an environment variable KUBECONFIG (pointing to a config file) as follows:

$ cat $KUBECONFIG
---
apiVersion: v1
clusters:
- name: gitlab-deploy
  cluster:
    server: https://KUBE01:6443
    certificate-authority-data: <CA_B64>
contexts:
- name: gitlab-deploy
  context:
    cluster: gitlab-deploy
    namespace: dev
    user: gitlab-deploy
current-context: gitlab-deploy
kind: Config
users:
- name: gitlab-deploy
  user:
    token: gitlab-deploy-token-<secret>

We can verify that kubectl is actually using the above gitlab-deploy context:

$ kubectl config current-context
gitlab-deploy

However, attempting to actually affect KUBE01 fails:

$ kubectl get pods
error: You must be logged in to the server (Unauthorized)

On my machine we can verify that the namespace and service account tokens are correct:

$ kubectl get sa/gitlab-deploy -o yaml --namespace dev
apiVersion: v1
kind: ServiceAccount
metadata:
  <snip metadata>
  name: gitlab-deploy
  namespace: dev
secrets:
- name: gitlab-deploy-token-<secret>

I cannot find any documentation about this other than it should just work, and all forum/stack exchange issues I've found about this error message are bad user/pass combinations; but as far as I can tell, my token, namespace, and cluster are all correct.

Answer 1

After a more detailed reading of the kuberneter authentication documentation and some trial and error, I have found the issue.

The "secret tokens" inside service account objects are not the actual secret tokens that we use for service account authentication; rather, it is a pointer to a secret object which, in turn, holds the real (bearer) token. It can be found as follows:

$ kubectl get secret gitlab-deploy-token-<secret> -o yaml
apiVersion: v1
data:
  ca.crt: <CA_B64>
  namespace: ZGV2
  token: <BEARER_TOKEN_B64>
kind: Secret
metadata:
  <snip metadata>
  name: gitlab-deploy-token-<secret>
  namespace: dev
type: kubernetes.io/service-account-token

The bearer token is, of course, base64 encoded due to it being a secret; but oddly enough, we don't base64 encode it in our KUBECONFIG file (like we do with e.g. the CA). Thus, what we had to do was find the above bearer token, decode it from base64, and add it as our token under the gitlab-deploy user. Then, authentication works.