Connect to container service clusterip from compute engine instance

Question

I have a few applications that run on regular Compute Engine nodes. In addition I have a Container Cluster that I am migrating applications to. Sooner or later all apps should be in Container Engine so service discovery is straight forward. But for now the apps on Compute Engine need to be able to talk to the Container Engine apps. The Container Engine apps are all registered as a service. For the sake of testing I used the "echoheaders" image:

$ kubectl describe svc echoheaders
Name:                   echoheaders
Namespace:              default
Labels:                 app=echoheaders
Selector:               app=echoheaders
Type:                   ClusterIP
IP:                     10.115.249.140
Port:                   http    80/TCP
Endpoints:              10.112.1.3:8080
Session Affinity:       None
No events.

The issue now is that I can only access the pod service from the Compute Engine node directly via 10.112.1.3:8080 but not via its clusterip 10.115.249.140:80. That only works from within the actual Compute Engine nodes.

I already tried to create a bastion route pointing to one of the Container Engine nodes but it still doesn't work:

$ gcloud compute routes describe gke-cluster-1-services 
creationTimestamp: '2016-04-05T05:39:55.275-07:00'
description: Route to Cluster-1 service IP range
destRange: 10.115.240.0/20
id: '926323215677918452'
kind: compute#route
name: gke-cluster-1-services
network: https://www.googleapis.com/compute/v1/projects/infrastructure-1173/global/networks/infra
nextHopInstance: https://www.googleapis.com/compute/v1/projects/infrastructure-1173/zones/europe-west1-d/instances/gke-cluster-1-5679a61a-node-f7iu
priority: 500
selfLink: https://www.googleapis.com/compute/v1/projects/infrastructure-1173/global/routes/gke-cluster-1-services

And on the firewall the Compute Node can connect to any.

Anybody happen to have pointers what could be missing to allow the Compute Engine nodes access the Compute Node Services by their ClusterIPs?

Thanks

Answer 1

Kubernetes expects anything within the cluster to be able to talk with everything else. GKE accomplishes this with advanced routing. By default, this lets GKE containers and GCE nodes on the same network communicate. This is why you could hit your containers directly.

A ClusterIP is only reachable within the Kubernetes cluster. These IPs are managed by iptables on just Kubernetes nodes. This is why you can't hit your service from the GCE nodes, but you can hit it from your containers.

Bastion routes send all traffic to the cluster's subnet to a cluster node. The node then routes the flow correctly. Create multiple bastion routes to multiple nodes at the same priority to avoid hotspotting a single node.

Try using the cluster's full /14, which you can find under the cluster's description in the container engine UI.