How can I debug why a Kubernetes load balancer service isn't responding on a port?

Question

I have set up a simple Kubernetes load balancer service in front of a Node.js container, which should be exposing port 80, but I can't get a response out of it. How can I debug how the load balancer is handling requests to port 80? Are there logs I can inspect?

I have set up a load balancer service and a replication controller as described in the Kubernetes guestbook example.

The service/load balancer spec is similar to this:

{
   "kind":"Service",
   "apiVersion":"v1",
   "metadata":{
      "name":"guestbook",
      "labels":{
         "app":"guestbook"
      }
   },
   "spec":{
      "ports": [
         {
           "port":3000,
           "targetPort":"http-server"
         }
      ],
      "selector":{
         "app":"guestbook"
      },
      "type": "LoadBalancer"
   }
}

As for my hosting platform, I'm using AWS and the OS is CoreOS alpha (976.0.0). Kubectl is at version 1.1.2.

Kubernetes Info

$ ~/.local/bin/kubectl --kubeconfig=/etc/kubernetes/kube.conf get pods
NAME            READY     STATUS    RESTARTS   AGE
busybox-sleep   1/1       Running   0          18m
web-s0s5w       1/1       Running   0          12h
$ ~/.local/bin/kubectl --kubeconfig=/etc/kubernetes/kube.conf get services
NAME         CLUSTER_IP   EXTERNAL_IP   PORT(S)   SELECTOR   AGE
kubernetes   10.3.0.1     <none>        443/TCP   <none>     1d
web          10.3.0.171

Answer 1

Here is the primary debugging document for Services:

http://kubernetes.io/docs/user-guide/debugging-services/

LoadBalancer creates an external resource. What exactly that resource is depends on your Cloud Provider - some of them don't support it at all (in this case, you might want to try NodePort instead).

Both Google and Amazon support external load balancers.

Overall, when asking these questions it's extremely helpful to know if you are running on Google Container Engine, Google Compute Engine, Amazon Web Services, Digital Ocean, Vagrant, or whatever, because the answer depends on that. Showing all your configs and all your existing Kubnernetes resources (kubectl get pods, kubectl get services) along with your Dockerfiles or which images you are using will also help.

For Google (GKE or GCE), you would verify the load balancer exists:

gcloud compute forwarding-rules list

The external load balancer will map port 80 to an arbitrary Node, but then the Kubernetes proxy will map that to an ephemeral port on the correct node that actually has a Pod with that label, then it will map to the container port. So you have to figure out which step along the way isn't working. Unfortunately all those kube-proxy and iptables jumps are quite difficult to follow, so usually I would first double check all my Pods exist and have labels that match the selector of the Service. I would double check that my container is exposing the right port, I am using the right name for the port, etc. You might want to create some other Pods that just make calls to the Service (using the environment variables or KubeDNS, see the Kubernetes service documentation if you don't know what I'm referring to) and verify it's accessible internally before debugging the load balancer.

Some other good debugging steps:

Verify that your Kubernetes Service exists:

kubectl get services
kubectl get pods

Check your logs of your pod

kubectl logs <pod name>

Check that your service is created internally by printing the environment variable for it

kubectl exec <pod name> -- printenv GUESTBOOK_SERVICE_HOST

try creating a new pod and see if the service can be reached internally through GUESTBOOK_SERVICE_HOST and GUESTBOOK_SERVICE_PORT.

kubectl describe pod <pod name>

will give the instance id of the pod, you can SSH to it and run Docker and verify your container is running, attach to it, etc. If you really want to get into the IP tables debugging, try

sudo iptables-save