kubectl port-forward connection refused [ socat ]

Question

I am running pyspark on one of the ports of kubernetes. I am trying to port forward to my local machine. I am getting this error while executing my python file.

Forwarding from 127.0.0.1:7077 -> 7077
Forwarding from [::1]:7077 -> 7077
Handling connection for 7077
E0401 01:08:11.964798   20399 portforward.go:400] an error occurred forwarding 7077 -> 7077: error forwarding port 7077 to pod 68ced395bd081247d1ee6b431776ac2bd3fbfda4d516da156959b6271c2ad90c, uid : exit status 1: 2019/03/31 19:38:11 socat[1748104] E connect(5, AF=2 127.0.0.1:7077, 16): Connection refused

this a few lines of my python file. I am getting error in the lines where conf is defined.

from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext

conf = SparkConf().setMaster("spark://localhost:7077").setAppName("Stand Alone Python Script")

I already tried installing socat on the kubernetes. I am using spark version 2.4.0 locally. I even tried exposing port 7077 in YAML file. Did not work out.

This is the YAML file used for deployment.

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  creationTimestamp: 2018-10-07T15:23:35Z
  generation: 16
  labels:
    chart: spark-0.2.1
    component: m3-zeppelin
    heritage: Tiller
    release: m3
  name: m3-zeppelin
  namespace: default
  resourceVersion: "55461362"
  selfLink: /apis/apps/v1beta1/namespaces/default/statefulsets/m3-zeppelin
  uid: f56e86fa-ca44-11e8-af6c-42010a8a00f2
spec:
  podManagementPolicy: OrderedReady
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      component: m3-zeppelin
  serviceName: m3-zeppelin
  template:
    metadata:
      creationTimestamp: null
      labels:
        chart: spark-0.2.1
        component: m3-zeppelin
        heritage: Tiller
        release: m3
    spec:
      containers:
      - args:
        - bash
        - -c
        - wget -qO- https://archive.apache.org/dist/spark/spark-2.2.2/spark-2.2.2-bin-hadoop2.7.tgz
          | tar xz; mv spark-2.2.2-bin-hadoop2.7 spark; curl -sSLO https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-latest-hadoop2.jar;
          mv gcs-connector-latest-hadoop2.jar lib; ./bin/zeppelin.sh
        env:
        - name: SPARK_MASTER
          value: spark://m3-master:7077
        image: apache/zeppelin:0.8.0
        imagePullPolicy: IfNotPresent
        name: m3-zeppelin
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        resources:
          requests:
            cpu: 100m
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /zeppelin/conf
          name: m3-zeppelin-config
        - mountPath: /zeppelin/notebook
          name: m3-zeppelin-notebook
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - metadata:
      creationTimestamp: null
      name: m3-zeppelin-config
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10G
      storageClassName: standard
    status:
      phase: Pending
  - metadata:
      creationTimestamp: null
      name: m3-zeppelin-notebook
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10G
      storageClassName: standard
    status:
      phase: Pending
status:
  collisionCount: 0
  currentReplicas: 1
  currentRevision: m3-zeppelin-5779b84d99
  observedGeneration: 16
  readyReplicas: 1
  replicas: 1
  updateRevision: m3-zeppelin-5779b84d99
  updatedReplicas: 1

Answer 1

Focusing specifically on the error from the Kubernetes perspective it could be related to:

• Mismatch between the ports that request is sent to and the receiving end (for example sending a request to an NGINX instance on port: 1234)
• Pod not listening on a desired port.

---

I've managed to reproduce this error with a Kubernetes cluster created with kubespray.

Assuming that you've run following steps:

• $ kubectl create deployment nginx --image=nginx
• $ kubectl port-forward deployment/nginx 8080:80

Everything should be correct and the NGINX welcome page should appear when running: $ curl localhost:8080.

If we made a change to the port-forward command like below (notice the 1234 port):

• $ kubectl port-forward deployment/nginx 8080:1234

You will get following error:

Forwarding from 127.0.0.1:8080 -> 1234
Forwarding from [::1]:8080 -> 1234
Handling connection for 8080
E0303 22:37:30.698827  625081 portforward.go:400] an error occurred forwarding 8080 -> 1234: error forwarding port 1234 to pod e535674b2c8fbf66252692b083f89e40f22e48b7a29dbb98495d8a15326cd4c4, uid : exit status 1: 2021/03/23 11:44:38 socat[674028] E connect(5, AF=2 127.0.0.1:1234, 16): Connection refused

This would also work on a Pod that application haven't bound to the port and/or is not listening.

A side note!

You can simulate it by running an Ubuntu Pod where you try to curl its port 80. It will fail as nothing listens on its port. Try to exec into it and run $ apt update && apt install -y nginx and try to curl again (with kubectl port-forward configured). It will work and won't produce the error mentioned (socat).

---

Addressing the part of the question:

I even tried exposing port 7077 in YAML file. Did not work out.

If you mean that you've included the - containerPort: 8080. This field is purely informational and does not carry any configuration to be made. You can read more about it here:

• Stackoverflow.com: Answer: Why do we need a port/containerPort in a Kuberntes deployment/container definition?

(which besides that I consider incorrect as you are using the port: 7077)

---

As for the $ kubectl port-forward --address 0.0.0.0. It's a way to expose your port-forward so that it would listen on all interfaces (on a host machine). It could allow for an access to your port-forward from LAN:

• $ kubectl port-forward --help (part):

  # Listen on port 8888 on localhost and selected IP, forwarding to 5000 in the pod
  kubectl port-forward --address localhost,10.19.21.23 pod/mypod 8888:5000

Additional resources:

• Kubernetes.io: Docs: Tasks: Access application cluster: Port forward access to application cluster