Menu
Yesterday, I built a full-featured example which uses Terraform to create a network and a GKE cluster in Google Cloud Platform. The whole thing runs in Vagrant on a CentOS 7 VM and installs both gcloud, kubectl, and helm. I also extended the example to use Helm to install Spinnaker.
The GKE cluster is called gke-test-1. In my documentation I documented getting kubectl setup:
gcloud container clusters get-credentials --region=us-west1 gke-test-1
After this, I was able to use various kubectl commands to get nodes, get pods, get services, and get deployments, as well as all other cluster management commands. I was able to also use Helm to install Tiller and ultimately deploy Spinnaker.
However, today, the same process doesn't work for me. I spun up the network, subnet, GKE cluster, and the node pool, and whenever I try to use commands to get various resoures, I get this response:
[vagrant@katyperry vagrant]$ kubectl get nodes
No resources found.
Error from server (NotAcceptable): unknown (get nodes)
[vagrant@katyperry vagrant]$ kubectl get pods
No resources found.
Error from server (NotAcceptable): unknown (get pods)
[vagrant@katyperry vagrant]$ kubectl get services
No resources found.
Error from server (NotAcceptable): unknown (get services)
[vagrant@katyperry vagrant]$ kubectl get deployments
No resources found.
Error from server (NotAcceptable): unknown (get deployments.extensions)
Interestingly enough, it seems that some command do work:
[vagrant@katyperry vagrant]$ kubectl describe nodes | head
Name: gke-gke-test-1-default-253fb645-scq8
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/fluentd-ds-ready=true
beta.kubernetes.io/instance-type=n1-standard-4
beta.kubernetes.io/os=linux
cloud.google.com/gke-nodepool=default
failure-domain.beta.kubernetes.io/region=us-west1
failure-domain.beta.kubernetes.io/zone=us-west1-b
kubernetes.io/hostname=gke-gke-test-1-default-253fb645-scq8
When I open a shell in Google Cloud console, after running the same login command, I'm able to use kubectl to do all of the above:
naftuli_kay@naftuli-test:~$ gcloud beta container clusters get-credentials gke-test-1 --region us-west1 --project naftuli-test
Fetching cluster endpoint and auth data.
kubeconfig entry generated for gke-test-1.
naftuli_kay@naftuli-test:~$ kubectl get pods
No resources found.
naftuli_kay@naftuli-test:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-gke-test-1-default-253fb645-scq8 Ready <none> 40m v1.8.10-gke.0
gke-gke-test-1-default-253fb645-tfns Ready <none> 40m v1.8.10-gke.0
gke-gke-test-1-default-8bf306fc-n8jz Ready <none> 40m v1.8.10-gke.0
gke-gke-test-1-default-8bf306fc-r0sq Ready <none> 40m v1.8.10-gke.0
gke-gke-test-1-default-aecb57ba-85p4 Ready <none> 40m v1.8.10-gke.0
gke-gke-test-1-default-aecb57ba-n7n3 Ready <none> 40m v1.8.10-gke.0
naftuli_kay@naftuli-test:~$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.0.64.1 <none> 443/TCP 43m
naftuli_kay@naftuli-test:~$ kubectl get deployments
No resources found.
The only difference I can see is the difference between the kubectl version; Vagrant has the latest version, 1.11.0, and the Google Cloud console has 1.9.7.
I will attempt to downgrade.
Is this a known issue and what, if anything, can I do to work around it?
EDIT: This is reproducible and I can't find a way to prevent it from recurring. I tore down all of my infrastructure and then stood it up again. The Terraform is available here.
After provisioning the resources, I waited until the cluster reported being healthy:
[vagrant@katyperry vagrant]$ gcloud container clusters describe \
--region=us-west1 gke-test-1 | grep -oP '(?<=^status:\s).*'
RUNNING
I then setup my login credentials:
[vagrant@katyperry vagrant]$ gcloud container clusters get-credentials \
--region=us-west1 gke-test-1
I again attempted to get nodes:
[vagrant@katyperry vagrant]$ kubectl get nodes
No resources found.
Error from server (NotAcceptable): unknown (get nodes)
The cluster appears green in the Google Cloud dashboard:
Apparently, this is a reproducible problem, as I'm able to recreate it using the same Terraform and commands.
335
After successfully reproducing the issue multiple times by destroying and recreating all the infrastructure, I found some arcane post on GitLab that mentions a Kubernetes GitHub issue that seems to indicate:
...in order to maintain compatibility with 1.8.x servers (which are within the supported version skew of +/- one version)
Emphasis on the "+/- one version."
Upgrading the masters and the workers to Kubernetes 1.10 seems to entirely have addressed the issue, as I can now list nodes and pods with impunity:
[vagrant@katyperry vagrant]$ kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.4-gke.2", GitCommit:"eb2e43842aaa21d6f0bb65d6adf5a84bbdc62eaf", GitTreeState:"clean", BuildDate:"2018-06-15T21:48:39Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}
[vagrant@katyperry vagrant]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-gke-test-1-default-5989a78d-dpk9 Ready <none> 42s v1.10.4-gke.2
gke-gke-test-1-default-5989a78d-kh9b Ready <none> 58s v1.10.4-gke.2
gke-gke-test-1-default-653ba633-091s Ready <none> 46s v1.10.4-gke.2
gke-gke-test-1-default-653ba633-4zqq Ready <none> 46s v1.10.4-gke.2
gke-gke-test-1-default-848661e8-cv53 Ready <none> 53s v1.10.4-gke.2
gke-gke-test-1-default-848661e8-vfr6 Ready <none> 52s v1.10.4-gke.2
It appears that Google Cloud Platform's cloud shell pins to kubectl 1.9, which is within the version gap supported by the ideas expressed above.
Thankfully, the Kubernetes RHEL repository has a bunch of versions to choose from so it's possible to pin:
[vagrant@katyperry gke]$ yum --showduplicates list kubectl
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.usc.edu
* epel: sjc.edge.kernel.org
* extras: mirror.sjc02.svwh.net
* updates: mirror.linuxfix.com
Installed Packages
kubectl.x86_64 1.11.0-0 @kubernetes
Available Packages
kubectl.x86_64 1.5.4-0 kubernetes
kubectl.x86_64 1.6.0-0 kubernetes
kubectl.x86_64 1.6.1-0 kubernetes
kubectl.x86_64 1.6.2-0 kubernetes
kubectl.x86_64 1.6.3-0 kubernetes
kubectl.x86_64 1.6.4-0 kubernetes
kubectl.x86_64 1.6.5-0 kubernetes
kubectl.x86_64 1.6.6-0 kubernetes
kubectl.x86_64 1.6.7-0 kubernetes
kubectl.x86_64 1.6.8-0 kubernetes
kubectl.x86_64 1.6.9-0 kubernetes
kubectl.x86_64 1.6.10-0 kubernetes
kubectl.x86_64 1.6.11-0 kubernetes
kubectl.x86_64 1.6.12-0 kubernetes
kubectl.x86_64 1.6.13-0 kubernetes
kubectl.x86_64 1.7.0-0 kubernetes
kubectl.x86_64 1.7.1-0 kubernetes
kubectl.x86_64 1.7.2-0 kubernetes
kubectl.x86_64 1.7.3-1 kubernetes
kubectl.x86_64 1.7.4-0 kubernetes
kubectl.x86_64 1.7.5-0 kubernetes
kubectl.x86_64 1.7.6-1 kubernetes
kubectl.x86_64 1.7.7-1 kubernetes
kubectl.x86_64 1.7.8-1 kubernetes
kubectl.x86_64 1.7.9-0 kubernetes
kubectl.x86_64 1.7.10-0 kubernetes
kubectl.x86_64 1.7.11-0 kubernetes
kubectl.x86_64 1.7.14-0 kubernetes
kubectl.x86_64 1.7.15-0 kubernetes
kubectl.x86_64 1.7.16-0 kubernetes
kubectl.x86_64 1.8.0-0 kubernetes
kubectl.x86_64 1.8.1-0 kubernetes
kubectl.x86_64 1.8.2-0 kubernetes
kubectl.x86_64 1.8.3-0 kubernetes
kubectl.x86_64 1.8.4-0 kubernetes
kubectl.x86_64 1.8.5-0 kubernetes
kubectl.x86_64 1.8.6-0 kubernetes
kubectl.x86_64 1.8.7-0 kubernetes
kubectl.x86_64 1.8.8-0 kubernetes
kubectl.x86_64 1.8.9-0 kubernetes
kubectl.x86_64 1.8.10-0 kubernetes
kubectl.x86_64 1.8.11-0 kubernetes
kubectl.x86_64 1.8.12-0 kubernetes
kubectl.x86_64 1.8.13-0 kubernetes
kubectl.x86_64 1.8.14-0 kubernetes
kubectl.x86_64 1.9.0-0 kubernetes
kubectl.x86_64 1.9.1-0 kubernetes
kubectl.x86_64 1.9.2-0 kubernetes
kubectl.x86_64 1.9.3-0 kubernetes
kubectl.x86_64 1.9.4-0 kubernetes
kubectl.x86_64 1.9.5-0 kubernetes
kubectl.x86_64 1.9.6-0 kubernetes
kubectl.x86_64 1.9.7-0 kubernetes
kubectl.x86_64 1.9.8-0 kubernetes
kubectl.x86_64 1.10.0-0 kubernetes
kubectl.x86_64 1.10.1-0 kubernetes
kubectl.x86_64 1.10.2-0 kubernetes
kubectl.x86_64 1.10.3-0 kubernetes
kubectl.x86_64 1.10.4-0 kubernetes
kubectl.x86_64 1.10.5-0 google-cloud-sdk
kubectl.x86_64 1.10.5-0 kubernetes
kubectl.x86_64 1.11.0-0 kubernetes
EDIT: I have found the actual pull request that mentions this incompatibility. I have also found buried in the release notes the following information:
kubectl: This client version requires the
apps/v1API, so it will not work against a cluster version older than v1.9.0. Note that kubectl only guarantees compatibility with clusters that are +/- [one] minor version away.
This entire problem was an incompatibility between kubectl 1.11 and Kubernetes 1.8.
147
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With