Logo Questions Linux Laravel Mysql Ubuntu Git Menu
HTML CSS JAVASCRIPT SQL PYTHON PHP BOOTSTRAP JAVA JQUERY R React Kotlin
 

kubernetes: cannot access NodePort from other machines

Tags:

kubernetes

I am trying to install kubernetes. There was a problem when checking the container executed on kubernetes. I set the type of service to Nodeport, but I could not access from a node other than the one on which the container is running. Want to make it accessible from other computers, please tell me where it is different. I tried externalIPs and LoadBarancer, but it was impossible.

Enviroment

  • OS:Ubuntu 16.04 LTS
  • Kubernetes:1.8
  • Docker:17.09.0-ce
  • etcd:3.2.8
  • flannel:0.9.0

Network

  • Physical:10.1.1.0/24
  • flannel:172.16.0.0/16
  • docker:192.168.0.0/16

Machines

  • Master Node(2nodes):10.1.1.24,10.1.1.25
  • Worker Node(2nodes):10.1.1.26,10.1.1.27

kubectl describe svc nginx-cluster

Name:              nginx-cluster
Namespace:         default
Labels:            app=nginx-demo
Annotations:       <none>
Selector:          app=nginx-demo
Type:              ClusterIP
IP:                172.16.236.159
Port:              <unset>  8090/TCP
TargetPort:        80/TCP
Endpoints:         192.168.24.2:80
Session Affinity:  None
Events:            <none>

kubectl describe svc nginx-service

Name:                     nginx-service
Namespace:                default
Labels:                   app=nginx-demo
Annotations:              <none>
Selector:                 app=nginx-demo
Type:                     NodePort
IP:                       172.16.199.69
Port:                     <unset>  8090/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  31659/TCP
Endpoints:                192.168.24.2:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

running container worker node(10.1.1.27)

curl 10.1.1.27:31659

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

worker node(10.1.1.26)

curl 10.1.1.27:31659

curl: (7) Failed to connect to 10.1.1.27 port 31659:Connection timed out.

other machine(10.1.1.XX)

curl 10.1.1.27:31659

curl: (7) Failed to connect to 10.1.1.27 port 31659:Connection timed out.

kubectl get pods -o wide

NAME                          READY     STATUS    RESTARTS   AGE       IP             NODE
echoserver-848b75d85-9fx7r    1/1       Running   3          6d        192.168.70.2   k8swrksv01
nginx-demo-85cc49574c-wv2b9   1/1       Running   3          6d        192.168.2.2    k8swrksv02

kubectl get svc -o wide

NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE       SELECTOR
clusterip       ClusterIP   172.16.39.77     <none>        80/TCP           6d        run=echoserver
kubernetes      ClusterIP   172.16.0.1       <none>        443/TCP          10d       <none>
nginx-cluster   ClusterIP   172.16.236.159   <none>        8090/TCP         6d        app=nginx-demo
nginx-service   NodePort    172.16.199.69    <none>        8090:31659/TCP   6d        app=nginx-demo
nodeport        NodePort    172.16.38.40     <none>        80:31317/TCP     6d        run=echoserver

netstat -ntlp

tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      1963/kubelet
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      2202/kube-proxy
tcp        0      0 127.0.0.1:4243          0.0.0.0:*               LISTEN      1758/dockerd
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      996/sshd
tcp6       0      0 :::4194                 :::*                    LISTEN      1963/kubelet
tcp6       0      0 :::10250                :::*                    LISTEN      1963/kubelet
tcp6       0      0 :::31659                :::*                    LISTEN      2202/kube-proxy
tcp6       0      0 :::10255                :::*                    LISTEN      1963/kubelet
tcp6       0      0 :::10256                :::*                    LISTEN      2202/kube-proxy
tcp6       0      0 :::31317                :::*                    LISTEN      2202/kube-proxy
tcp6       0      0 :::22                   :::*                    LISTEN      996/sshd

iptables-save

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-AZ4EGFEAU4RTSLJO - [0:0]
:KUBE-SEP-C7HQKKO26GIFOZZM - [0:0]
:KUBE-SEP-EWKNS2YCPXGJCXDC - [0:0]
:KUBE-SEP-LQVPUPFGW6BWATIP - [0:0]
:KUBE-SEP-OMMOFZ27GPKZ4OPA - [0:0]
:KUBE-SEP-UD3HOGDD5NDLNY74 - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-CQNAS6RSUGJF2C2D - [0:0]
:KUBE-SVC-GKN7Y2BSGW4NJTYL - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-XP7QDA4CRQ2QA33W - [0:0]
:KUBE-SVC-Z5P6OMNAEVLAQUTS - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 192.168.2.0/24 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 192.168.0.0/16 -d 192.168.0.0/16 -j RETURN
-A POSTROUTING -s 192.168.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
-A POSTROUTING ! -s 192.168.0.0/16 -d 192.168.2.0/24 -j RETURN
-A POSTROUTING ! -s 192.168.0.0/16 -d 192.168.0.0/16 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx-service:" -m tcp --dport 31659 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx-service:" -m tcp --dport 31659 -j KUBE-SVC-GKN7Y2BSGW4NJTYL
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport:" -m tcp --dport 31317 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport:" -m tcp --dport 31317 -j KUBE-SVC-XP7QDA4CRQ2QA33W
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-AZ4EGFEAU4RTSLJO -s 192.168.70.2/32 -m comment --comment "default/clusterip:" -j KUBE-MARK-MASQ
-A KUBE-SEP-AZ4EGFEAU4RTSLJO -p tcp -m comment --comment "default/clusterip:" -m tcp -j DNAT --to-destination 192.168.70.2:8080
-A KUBE-SEP-C7HQKKO26GIFOZZM -s 192.168.70.2/32 -m comment --comment "default/nodeport:" -j KUBE-MARK-MASQ
-A KUBE-SEP-C7HQKKO26GIFOZZM -p tcp -m comment --comment "default/nodeport:" -m tcp -j DNAT --to-destination 192.168.70.2:8080
-A KUBE-SEP-EWKNS2YCPXGJCXDC -s 10.1.1.25/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-EWKNS2YCPXGJCXDC -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-EWKNS2YCPXGJCXDC --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.1.1.25:6443
-A KUBE-SEP-LQVPUPFGW6BWATIP -s 192.168.2.2/32 -m comment --comment "default/nginx-service:" -j KUBE-MARK-MASQ
-A KUBE-SEP-LQVPUPFGW6BWATIP -p tcp -m comment --comment "default/nginx-service:" -m tcp -j DNAT --to-destination 192.168.2.2:80
-A KUBE-SEP-OMMOFZ27GPKZ4OPA -s 10.1.1.24/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-OMMOFZ27GPKZ4OPA -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-OMMOFZ27GPKZ4OPA --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.1.1.24:6443
-A KUBE-SEP-UD3HOGDD5NDLNY74 -s 192.168.2.2/32 -m comment --comment "default/nginx-cluster:" -j KUBE-MARK-MASQ
-A KUBE-SEP-UD3HOGDD5NDLNY74 -p tcp -m comment --comment "default/nginx-cluster:" -m tcp -j DNAT --to-destination 192.168.2.2:80
-A KUBE-SERVICES -d 172.16.236.159/32 -p tcp -m comment --comment "default/nginx-cluster: cluster IP" -m tcp --dport 8090 -j KUBE-SVC-Z5P6OMNAEVLAQUTS
-A KUBE-SERVICES -d 172.16.199.69/32 -p tcp -m comment --comment "default/nginx-service: cluster IP" -m tcp --dport 8090 -j KUBE-SVC-GKN7Y2BSGW4NJTYL
-A KUBE-SERVICES -d 172.16.38.40/32 -p tcp -m comment --comment "default/nodeport: cluster IP" -m tcp --dport 80 -j KUBE-SVC-XP7QDA4CRQ2QA33W
-A KUBE-SERVICES -d 172.16.39.77/32 -p tcp -m comment --comment "default/clusterip: cluster IP" -m tcp --dport 80 -j KUBE-SVC-CQNAS6RSUGJF2C2D
-A KUBE-SERVICES -d 172.16.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-CQNAS6RSUGJF2C2D -m comment --comment "default/clusterip:" -j KUBE-SEP-AZ4EGFEAU4RTSLJO
-A KUBE-SVC-GKN7Y2BSGW4NJTYL -m comment --comment "default/nginx-service:" -j KUBE-SEP-LQVPUPFGW6BWATIP
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-OMMOFZ27GPKZ4OPA --mask 255.255.255.255 --rsource -j KUBE-SEP-OMMOFZ27GPKZ4OPA
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-EWKNS2YCPXGJCXDC --mask 255.255.255.255 --rsource -j KUBE-SEP-EWKNS2YCPXGJCXDC
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-OMMOFZ27GPKZ4OPA
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-EWKNS2YCPXGJCXDC
-A KUBE-SVC-XP7QDA4CRQ2QA33W -m comment --comment "default/nodeport:" -j KUBE-SEP-C7HQKKO26GIFOZZM
-A KUBE-SVC-Z5P6OMNAEVLAQUTS -m comment --comment "default/nginx-cluster:" -j KUBE-SEP-UD3HOGDD5NDLNY74
COMMIT
*filter
:INPUT ACCEPT [40:14606]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [42:6275]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:DOCKER-USER - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION -j RETURN
-A DOCKER-USER -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
COMMIT
like image 961
Y.Yama Avatar asked Oct 10 '17 13:10

Y.Yama

Video Answer

1 Answers

additional information. Something is supposed to block the port, but that is unknown...

pod running node

nmap 10.1.1.27 -p31000-32000

Not shown: 999 closed ports
PORT      STATE SERVICE
31317/tcp open  unknown
31659/tcp open  unknown

other node

nmap 10.1.1.27 -p31000-32000

Not shown: 999 closed ports
PORT      STATE    SERVICE
31317/tcp filtered unknown
31659/tcp filtered unknown
like image 88
Y.Yama Avatar answered Sep 28 '22 19:09

Y.Yama
Sign in to Comment

Related questions

Windows & Linux living together in Kubernetes cluster
Command not found error when initializing the tiller using `helm init`
What are generators in kubernetes kubectl?
Kubernetes autoscaler - NotTriggerScaleUp' pod didn't trigger scale-up (it wouldn't fit if a new node is added)
Openshift: No nodes are available that match all of the following predicates:: Insufficient cpu (173), MatchNodeSelector (5)
Where can i get the API_KEY in Kubernetes installation
How do I use kubectl via an ssh tunnel to access my internal ELB over my kubernetes API?
Accessing service using istio ingress gives 503 error when mTLS is enabled
Docker entrypoint permission denied
Helm Test best practices
Minikube with ingress example not working
Worker start to fail CSINodeIfo: error updating CSINode annotation
Kubernetes Jenkins plugin - slaves always offline
How to get kube-dns working in Vagrant cluster using kubeadm and Weave
kubeadm: master node never ready
minikube service %servicename% --url return nothing
storageclass.storage.k8s.io "standard" not found for pvc on bare metal Kubernetes cluster
Ingress is not getting address on GKE and GCE
How to change running pods limits in Kubernetes?
Kubernetes: VPN server and DNS issues

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!