Trouble accessing Kubernetes endpoints

Question

I'm bringing up Spark on Kubernetes according to this example: https://github.com/kubernetes/kubernetes/tree/master/examples/spark

For some reason, I'm having problems getting the master to listen on :7077 for connections from worker nodes. It appears that connections aren't being proxied down from the service. If I bring the service up, then bring the master controller up with the $SPARK_MASTER_IP set to spark-master, it correctly resolves to the service IP but cannot bind the port. If I set the ip to localhost instead, it binds a local port and comes up -- since services should forward socket connections down to the pod endpoint this should be fine, so we move on.

Now I bring up workers. They attempt to connect to the service IP on :7077 and cannot. It seems as if connections to the service aren't making it down to the endpoint. Except...

I also have a webui service configured as in the example. If I connect to it with kubectl --proxy I can get down to the web service that's served on :8080 from spark-master, by hitting it through the webui service. Yet the nearly identically-configured spark-master service on port 7077 gives no love. If I configure the master to bind a local IP, it comes up but doesn't get connections from the service. If I configure it to bind through the service, the bind fails and it can't come up at all.

I'm running out of ideas as to why this might be happening -- any assistance is appreciated. I'm happy to furnish more debugging info on request.

Answer 1

I'm sorry, the Spark example was broken, in multiple ways.

The issue: https://github.com/kubernetes/kubernetes/issues/17463

It now works, as of 2/25/2016, and is passing our continuous testing, at least at HEAD (and the next Kubernetes 1.2 release).

Note that DNS is required, though it is set up by default in a number of cloud provider implementations, including GCE and GKE.