Connection refused to GCP LoadBalancer in Kubernetes

Question

When I create a deployment and a service in a Kubernetes Engine in GCP I get connection refused for no apparent reason.

The service creates a Load Balancer in GCP and all corresponding firewall rules are in place (allows traffic to port 80 from 0.0.0.0/0). The underlying service is running fine, when I kubectl exec into the pod and curl localhost:8000/ I get the correct response.

This deployment setting used to work just fine for other images, but yesterday and today I keep getting

curl: (7) Failed to connect to 35.x.x.x port 80: Connection refused

What could be the issue? I tried deleting and recreating the service multiple times, with no luck.

kind: Service
apiVersion: v1
metadata:
  name: my-app
spec:
  selector:
    app: app 
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8000
---
apiVersion: apps/v1beta2 
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app
  template:
    metadata:
      labels:
        app: app
    spec:
      containers:
      - name: my-app
        image: gcr.io/myproject/my-app:0.0.1
        imagePullPolicy: Always
        ports:
        - containerPort: 8000

Answer 1

This turned out to be a dumb mistake on my part. The gunicorn server was using a bind to 127.0.0.1 instead of 0.0.0.0, so it wasn't accessible from outside of the pod, but worked when I exec-ed into the pod.

The fix in my case was changing the entrypoint of the Dockerfile to

CMD [ "gunicorn", "server:app", "-b", "0.0.0.0:8000", "-w", "3" ]

rebuilding the image and updating the deployment.