Kubernetes service discovery doesn't resolve service host on minikube

Question

I'm trying to run two services with k8s running on minikube installation. So I have the sevice Foo and Bar, so when I'm accessing the service Foo it must trigger service Bar to get the data.

Here is the deployment configuration for Foo service foo-deployment.yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: foo-server
  labels:
    app: foo-server
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: foo-server
        serving: "true"
    spec:
      containers:
      - name: foo-server
        image: foo-container
        env:
        - name: BAR_SERVICE_URL
          value: http://bar-server:8081
        ports:
        - containerPort: 8080
          name: http

foo-svc.yaml:

apiVersion: v1
kind: Service
metadata:
  name: foo-server
  labels:
    app: foo-server
spec:
  type: NodePort
  ports:
  - port: 8080
    targetPort: 8080
    name: http
  selector:
    app: foo-server
    serving: "true"

The BAR_SERVICE_URL evn variable is used inside the Foo so the service knows the host to Bar. And here is the service configuration for Bar service bar-svc.yaml:

apiVersion: v1
kind: Service
metadata:
  name: bar-server
  labels:
    app: bar-server
spec:
  type: NodePort
  ports:
  - port: 8081
    targetPort: 8081
    name: http
  selector:
    app: bar-server

bar-deployment.yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: bar-server-v1
  labels:
    app: bar-server
    version: "1.0"
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: bar-server
        version: "1.0"
        serving: "true"
    spec:
      containers:
      - name: bar-server
        image: bar-container
        ports:
        - containerPort: 8081
          name: http

When I'm accessing the Foo service with the minikube ip and exposed service port I receive the following error in the pod logs:

2018-02-08 14:32:25.875 ERROR 1 --- [nio-8080-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://bar-server:8081/endpoint": bar-server; nested exception is java.net.UnknownHostException: bar-server] with root cause

java.net.UnknownHostException: bar-server

However, I can access the bar-server/endpoint using the minikube ip and exposed port like this http://192.168.99.100:31168/endpoint.

So looks like I can access the Foo and Bar services from outside the cluster. But Foo service cannot resolve the Bar service host and cannot access it.

UPD: Here are some update with troubleshouting details.

• On minikube the kube-dns addon is enabled.
• Trying from the iside of Foo pod to wget to the Bar service endpoint. I'm able to do it via IP address of the bar-server but not with bar-server:8081/endpoint.

• Trying nslookup kubernetes.default.svc.cluster.local localhost looks good. Here is the ouptput:

  Server: 127.0.0.1 Address 1: 127.0.0.1 localhost

• Also dns pod looks up and running. Here is the output for kubectl get pods -n=kube-system | grep -i dns

  kube-dns-6777479f6b-vxd7m 3/3 Running 9 1d

Answer 1

Kubernetes does service discovery using DNS (kube-dns). If DNS in k8s is working, service Foo should be able to ping or curl service Bar, and vice-versa.

A way to check if kube-dns is working is to see if the following commands work inside any pod:

nslookup kubernetes.default.svc.cluster.local localhost
nslookup kubernetes.default localhost

kube-dns pod's health can be checked in the output of:

kubectl get pods -n=kube-system | grep -i dns