How to expose kubernetes service to public without hardcoding to minion IP?

Question

I have a kubernetes cluster running with 2 minions. Currently I make my service accessible in 2 steps:

1. Start replication controller & pod
2. Get minion IP (using kubectl get minions) and set it as publicIPs for the Service.

What is the suggested practice for exposing service to the public? My approach seems wrong because I hard-code the IP-s of individual minion IP-s. It also seems to bypass load balancing capabilities of kubernetes services because clients would have to access services running on individual minions directly.

To set up the replication controller & pod I use:

id: frontend-controller
kind: ReplicationController
apiVersion: v1beta1
desiredState:
  replicas: 2
  replicaSelector:
    name: frontend-pod
  podTemplate:
    desiredState:
      manifest:
        version: v1beta1
        id: frontend-pod
        containers:
          - name: sinatra-docker-demo
            image: madisn/sinatra_docker_demo
            ports:
              - name: http-server
                containerPort: 4567
    labels:
      name: frontend-pod

To set up the service (after getting minion ip-s):

kind: Service
id: frontend-service
apiVersion: v1beta1
port: 8000
containerPort: http-server
selector:
  name: frontend-pod
labels:
  name: frontend
publicIPs: [10.245.1.3, 10.245.1.4]

Answer 1

As I mentioned in the comment above, the createExternalLoadBalancer is the appropriate abstraction that you are looking for, but unfortunately it isn't yet implemented for all cloud providers, and in particular for vagrant, which you are using locally.

One option would be to use the public IPs for all minions in your cluster for all of the services you want to be externalized. The traffic destined for the service will end up on one of the minions, where it will be intercepted by the kube-proxy process and redirected to a pod that matches the label selector for the service. This could result in an extra hop across the network (if you land on a node that doesn't have the pod running locally) but for applications that aren't extremely sensitive to network latency this will probably not be noticeable.

Answer 2

As Robert said in his reply this is something that is coming up, but unfortunately isn't available yet.

I am currently running a Kubernetes cluster on our datacenter network. I have 1 master and 3 minions all running on CentOS 7 virtuals (vcenter). The way I handled this was to create a dedicated "kube-proxy" server. I basically am just running the Kube-Proxy service (along with Flannel for networking) and then assigning "public" IP addresses to the network adapter attached to this server. When I say public I mean addresses on our local datacenter network. Then when I create a service that I would like to access outside of the cluster I just set the publicIPs value to one of the available IP addresses on the kube-proxy server. When someone or something attempts to connect to this service from outside the cluster it will hit the kube-proxy and then be redirected to the proper minion.

While this might seem like a work around, this is actually similar to what I would expect to be happening once they come up with a built in solution to this issue.

Answer 3

If you're running a cluster locally, a solution I used was to expose the service on your kubernetes nodes using the nodeport directive in your service definition and then round robin to every node in your cluster with HAproxy.

Here's what exposing the nodeport looks like:

apiVersion: v1
kind: Service
metadata:
  name: nginx-s
  labels:
    name: nginx-s
spec:
  type: NodePort
  ports:
    # must match the port your container is on in your replication controller
    - port: 80
      nodePort: 30000
  selector:
    name: nginx-s

Note: the value you specify must be within the configured range for node ports. (default: 30000-32767)

This exposes the service on the given nodeport on every node in your cluster. Then I set up a separate machine on the internal network running haproxy and a firewall that's reachable externally on the specified nodeport(s) you want to expose.

If you look at your nat table on one of your hosts, you can see what it's doing.

root@kube01:~# kubectl create -f nginx-s.yaml
You have exposed your service on an external port on all nodes in your
cluster.  If you want to expose this service to the external internet, you may
need to set up firewall rules for the service port(s) (tcp:30000) to serve traffic.

See http://releases.k8s.io/HEAD/docs/user-guide/services-firewalls.md for more details.
services/nginx-s
root@kube01:~# iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
KUBE-PORTALS-CONTAINER  all  --  anywhere             anywhere             /* handle ClusterIPs; NOTE: this must be before the NodePort rules */
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL
KUBE-NODEPORT-CONTAINER  all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL /* handle service NodePorts; NOTE: this must be the last rule in the chain */

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
KUBE-PORTALS-HOST  all  --  anywhere             anywhere             /* handle ClusterIPs; NOTE: this must be before the NodePort rules */
DOCKER     all  --  anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL
KUBE-NODEPORT-HOST  all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL /* handle service NodePorts; NOTE: this must be the last rule in the chain */

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  172.17.0.0/16        anywhere

Chain DOCKER (2 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Chain KUBE-NODEPORT-CONTAINER (1 references)
target     prot opt source               destination
REDIRECT   tcp  --  anywhere             anywhere             /* default/nginx-s: */ tcp dpt:30000 redir ports 42422

Chain KUBE-NODEPORT-HOST (1 references)
target     prot opt source               destination
DNAT       tcp  --  anywhere             anywhere             /* default/nginx-s: */ tcp dpt:30000 to:169.55.21.75:42422

Chain KUBE-PORTALS-CONTAINER (1 references)
target     prot opt source               destination
REDIRECT   tcp  --  anywhere             192.168.3.1          /* default/kubernetes: */ tcp dpt:https redir ports 51751
REDIRECT   tcp  --  anywhere             192.168.3.192        /* default/nginx-s: */ tcp dpt:http redir ports 42422

Chain KUBE-PORTALS-HOST (1 references)
target     prot opt source               destination
DNAT       tcp  --  anywhere             192.168.3.1          /* default/kubernetes: */ tcp dpt:https to:169.55.21.75:51751
DNAT       tcp  --  anywhere             192.168.3.192        /* default/nginx-s: */ tcp dpt:http to:169.55.21.75:42422
root@kube01:~#

Particularly this line

DNAT       tcp  --  anywhere             anywhere             /* default/nginx-s: */ tcp dpt:30000 to:169.55.21.75:42422

And finally, if you look at netstat, you can see kube-proxy is listening and waiting for that service on that port.

root@kube01:~# netstat -tupan | grep 42422
tcp6       0      0 :::42422                :::*                    LISTEN      20748/kube-proxy
root@kube01:~#

Kube-proxy will listen on a port for each service, and do network address translation into your virtual subnet that your containers reside in. (I think?) I used flannel.

---

For a two node cluster, that HAproxy configuration might look similiar to this:

listen sampleservice 0.0.0.0:80
    mode http
    stats enable
    balance roundrobin
    option httpclose
    option forwardfor
    server noname 10.120.216.196:30000 check
    server noname 10.155.236.122:30000 check
    option httpchk HEAD /index.html HTTP/1.0

And your service is now reachable on port 80 via HAproxy. If any of your nodes go down, the containers will be moved to another node thanks to replication controllers and HAproxy will only route to your nodes that are alive.

I'm curious what methods others have used though, that's just what I came up with. I don't usually post on stack overflow, so apologies if I'm not following conventions or proper formatting.