kubernetes error syncing pod - how to debug

Question

I often get errors about pods not syncing / ImagePullBackOff errors on my Kubernetes cluster on Google Kubernetes Engine. But I'm not sure how to debug the issue as I can't establish the root cause.

In the Google dashboard I can see the ReplicaSet has the warning: Pod errors: ImagePullBackOff

If I drill down to the pod, I can see in the logs: Message: Error syncing pod Reason: FailedSync

but nothing further. Why would a pod fail to sync?

UPDATE: further down in the logs I see:

Failed to pull image "cockroachdb/cockroach:v1.1.3": rpc error: code = Unknown desc = failed to register layer: ApplyLayer exit status 1 stdout: stderr: open /usr/share/zoneinfo/right/America/Pangnirtung: no space left on device

I've allocated cockroachdb 1 GB of persistent storage - going to try to increase to 10 GB to see if that fixes anything. Or do I need to increase the disk size on the node pools?

Answer 1

ImagePullBackOff occurs most of the time due to typos in the image name or not being able to reach the repository:

• Check for typos by copy/pasting the image name in a docker pull command (copy/paste so that any error is also copy/pasted: You want to find the error and not confirm your own bias).
• Check for reachability of DNS by logging into a pod and executing a nslookup/dig command (or ping opr anything which hits the DNS).