Exposing a DaemonSet service for consumption by pods on the same node

Question

I'm trying to install a service mesh in my Kubernetes cluster, and for that I want to accomplish the following:

• The service mesh pods are run as a Daemon Set, with one replica on each node.
• The service mesh is not directly reachable from outside the cluster.
• The application pods use the service mesh pod on the same node as a http proxy for all requests.

In order to do so, I've added the following configuration to the container spec on the application Deployment:

# deployment.spec.template.spec.containers[0]
  env:
  - name: NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName
  - name: HTTP_PROXY
    value: http://$(NODE_NAME):4140

However, I'm failing to expose the service mesh properly; I've tried with both services of type: ClusterIP and type: LoadBalancer, but with the former I fail to connect, and with the latter I expose the service mesh publicly, which I don't want to do. I've also tried to get something working with NodePort, but I haven't even been able to figure out a valid configuration there yet (and there seems to be some opinion that NodePort should be avoided if possible...).

How should I expose the service mesh, preferrably on ports 4140-4142 and 9990, in a way that is reachable on a specific node (the current one) from other pods?

Answer 1

You could put your pods in the host network space. That way you won't need any Service and your Pod will be reachable on the port you declare in the PodSpec on every node.

You could avoid external reachability by binding your service to 127.0.0.1 rather than 0.0.0.0

If you want to give it a try, you can run a port in the host network space by adding the following to the PodSpec of your DaemonSet:

hostNetwork: true

Please be aware that, with this solution, you'll need to use the host's IP address in order to connect to your pod.

In order to get internal DNS names resolution in your hostNetworked pods, you also need to set DNS policy like this:

dnsPolicy: ClusterFirstWithHostNet

This will ensure your pods will use the internal DNS server for name resolution.