Can't Access Kubernetes Service Exposed via NodePort

Question

I'm using minikube to test kubernetes on latest MacOS.

Here are my relevant YAMLs:

namespace.yml

apiVersion: v1
kind: Namespace
metadata:
  name: micro
  labels:
    name: micro

deployment.yml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: adderservice
spec:
  replicas: 1
  template:
    metadata:
      labels:
        run: adderservice
    spec:
      containers:
      - name: adderservice
        image: jeromesoung/adderservice:0.0.1
        ports:
        - containerPort: 8080

service.yml

apiVersion: v1
kind: Service
metadata:
  name: adderservice
  labels:
    run: adderservice
spec:
  ports:
    - port: 8080
      name: main
      protocol: TCP
      targetPort: 8080
  selector:
    run: adderservice
  type: NodePort

After running minikube start, the steps I took to deploy is as follows:

1. kubectl create -f namespace.yml to create the namespace

2. kubectl config set-context minikube --namespace=micro

3. kubectl create -f deployment.yml

4. kubectl create -f service.yml

Then, I get the NodeIP and NodePort with below commands:

1. kubectl get services to get the NodePort

$ kubectl get services
NAME           TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
adderservice   NodePort   10.99.155.255   <none>        8080:30981/TCP   21h

2. minikube ip to get the nodeIP

$ minikube ip
192.168.99.103

But when I do curl, I always get Connection Refused like this:

$ curl http://192.168.99.103:30981/add/1/2
curl: (7) Failed to connect to 192.168.99.103 port 30981: Connection refused

So I checked node, pod, deployment and endpoint as follows:

$ kubectl get nodes
NAME       STATUS    ROLES     AGE       VERSION
minikube   Ready     master    23h       v1.13.3

$ kubectl get pods
NAME                            READY     STATUS    RESTARTS   AGE
adderservice-5b567df95f-9rrln   1/1       Running   0          23h

$ kubectl get deployments
NAME           DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
adderservice   1         1         1            1           23h

$ kubectl get endpoints
NAME           ENDPOINTS         AGE
adderservice   172.17.0.5:8080   21h

I also checked service list from minikube with:

$ minikube service -n micro adderservice --url
http://192.168.99.103:30981

I've read many posts regarding accessing k8s service via NodePorts. To my knowledge, I should be able to access the app with no problem. The only thing I suspect is that I'm using a custom namespace. Will this cause the access issue?

I know namespace will change the DNS, so, to be complete, I ran below commands also:

$ kubectl exec -ti adderservice-5b567df95f-9rrln -- nslookup kubernetes.default
Server:     10.96.0.10
Address:    10.96.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

$ kubectl exec -ti adderservice-5b567df95f-9rrln -- nslookup kubernetes.micro
Server:     10.96.0.10
Address:    10.96.0.10#53

Non-authoritative answer:
Name:   kubernetes.micro
Address: 198.105.244.130
Name:   kubernetes.micro
Address: 104.239.207.44

Could anyone help me out? Thank you.

Answer 1

The error Connection Refused mostly means that the application inside the container does not accept requests on the targeted interface or not mapped through the expected ports.

Things you need to be aware of:

• Make sure that your application bind to 0.0.0.0 so it can receive requests from outside the container either externally as in public or through other containers.
• Make sure that your application is actually listening on the containerPort and targetPort as expect

In your case you have to make sure that ADDERSERVICE_SERVICE_HOST equals to 0.0.0.0 and ADDERSERVICE_SERVICE_PORT equals to 8080 which should be the same value as targetPort in service.yml and containerPort in deployment.yml

Answer 2

Not answering the question but if someone who googled comes here like me who faced the same issue. Here is my solution for the same problem.

My Mac System IP and minikube IP are different.

So localhost:port didn't work instead try getting IP

minikube ip

Later, use that IP:Port to access the app and it works.

Answer 3

Check if service is really listening on 8080.

Try telnet within the container.

telnet 127.0.0.1 8080
.
.
.
telnet 172.17.0.5 8080