How to easily switch gcloud / kubectl credentials

Question

At work we use Kubernetes hosted in GCP. I also have a side project hosted in my personal GCP account using Google App Engine (deploy using gcloud app deploy).

Often when I try to run a command such as kubectl logs -f service-name, I get an error like "Error from server (Forbidden): pods is forbidden: User "[email protected]" cannot list resource "pods" in API group "" in the namespace "WORK_NAMESPACE": Required "container.pods.list" permission." and then I have to fight with kubectl for hours trying to get it to work.

Can somebody please break it down for a slow person like me, how gcloud and kubectl work together, and how I can easily switch accounts so I can use gcloud commands for my personal projects and kubectl commands for my work projects? I'm happy to nuke my whole config and start from scratch if that's what it takes. I've found various kubectl and gcloud documentation but it doesn't make much sense or talks in circles.

Edit: this is on Linux.

Answer 1

Had the same problem and doing all of the:

gcloud auth login
gcloud auth list
gcloud config set account
gcloud projects list

didn't help. I knew gcloud switched fine as I was able to list other resources with it directly. But it seems kubectl can't pick those changes up automatically, as kubectl/gcloud integration relies on the pre-generated key, which has a 1h expiration(not sure if it's a default but it's what it is on my machine right now). So, on top of setting right user/project/account with gcloud, you should re-generate the creds:

gcloud container clusters get-credentials <my-cluster> --zone <clusters-zone>

Answer 2

TL;DR

Edit: Kubernetes 1.26 has a change relating to this answer, now use gcloud auth to manage your different profiles.

1. Use gcloud auth to manage your separate profiles with the Google Cloud Platform.
2. Add an explicit account to use when requesting a token with the gke-gcloud-auth-plugin to your kubectl config file (~/.kube/config) to teach kubectl to use a specific account with a certain cluster rather than your active account.

# file: ~/.kube/config
users:
- name: gke_project-name_cluster-zone_cluster-name
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      command: gke-gcloud-auth-plugin
      args:
      - [email protected] # <<< Add this line

---

Can somebody please break it down for a slow person like me, how gcloud and kubectl work together, and how I can easily switch accounts so I can use gcloud commands for my personal projects and kubectl commands for my work projects?

Sure! By following Google's suggested instructions that lead to running gcloud container clusters get-credentials ... when configuring a kubernetes cluster, you will end up with a section of your kubeconfig that contains information on what kubectl should do to acquire an access token when communicating with a cluster that is configured with a given user. That will look something like this:

users:
- name: gke_project-name_cluster-zone_cluster-name
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args: null
      command: gke-gcloud-auth-plugin
      env: null
      installHint: Install gke-gcloud-auth-plugin for use with kubectl by following
        https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
      interactiveMode: IfAvailable
      provideClusterInfo: true

Basically, this tells kubectl to run gke-gcloud-auth-plugin when it needs a new token, and that the access token can be parsed from the output of that command. This is the crux in understanding how kubectl communicates with gcloud.

Like you, I use google cloud both personally and at work. The issue is that this user configuration block does not take into account the fact that it shouldn't use the currently active gcloud account when generating a credential. Even if you don't use kubernetes in either one of your two projects, extensions in vscode for example might try to run a kubectl command when you're working on something in a different project. If this were to happen after your current token is expired, gke-gcloud-auth-plugin might get invoked to generate a token using a personal account.

To prevent this from happening, I suggest using gcloud config configuations. Configurations are global configuration profiles that you can quickly switch between. They can store more than just account information, any gcloud config ... can be saved. For example, I have two configurations that look like this:

> gcloud config configurations list                                               

NAME      IS_ACTIVE  ACCOUNT             PROJECT            COMPUTE_DEFAULT_ZONE       COMPUTE_DEFAULT_REGION
work      False      [email protected]      work-project       us-west1-a                 us-west1
personal  True       [email protected]  personal-project   northamerica-northeast1-a  northamerica-northeast1

When setting up multiple configurations, you will have to run gcloud auth login with your separate accounts. We just need to instruct kubectl to use a specific account to talk to a given cluster, rather than whichever is active.

So then it boils down to this: if you set up your kubeconfig to execute gke-gcloud-auth-plugin [email protected] whenever kubectl needs a new access token for your personal cluster, it will use the account from your work configuration regardless of which account is currently active with the gcloud tool. That would then look like this:

# file: ~/.kube/config
users:
- name: gke_project-name_cluster-zone_cluster-name
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1

      # ----- Remove this ------
      args: null
      # -------------------------
      # +++++ Then add this: ++++
      args:
      - [email protected]
      # +++++++++++++++++++++++++

      command: gke-gcloud-auth-plugin
      env: null
      installHint: Install gke-gcloud-auth-plugin for use with kubectl by following
        https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
      interactiveMode: IfAvailable
      provideClusterInfo: true

Prior to Kubernetes 1.26, the process is the same, except you would use a gcloud config configuration directly, rather than an account, so:

1. Use gcloud config configurations to manage your separate profiles with Google Cloud Platform.

2. Find the cmd-args of your kubeconfig's user, then add an explicit --configuration argument to prevent gcloud from producing an access token for an unrelated profile.

   # file: ~/.kube/config
   users:
   - user:
       auth-provider:
         config:
           # !! Prior to GKE 1.26
           cmd-args: config --configuration=work config-helper --format=json