Kubernetes ingress-nginx gives 502 error (Bad Gateway)

Question

I have an EKS cluster for which I want : - 1 Load Balancer per cluster, - Ingress rules to direct to the right namespace and the right service.

I have been following this guide : https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes

My deployments:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello-world
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      containers:
      - name: hello-world
        image: IMAGENAME
        ports:
        - containerPort: 8000
          name: hello-world


---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bleble
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: bleble
  template:
    metadata:
      labels:
        app: bleble
    spec:
      containers:
      - name: bleble
        image: IMAGENAME
        ports:
        - containerPort: 8000
          name: bleble

the service of those deployments:

apiVersion: v1
kind: Service
metadata: 
  name: hello-world-svc
spec: 
  ports: 
     -  port: 8080
        protocol: TCP
        targetPort: 8000
  selector: 
    app: hello-world
  type: NodePort

---

apiVersion: v1
kind: Service
metadata: 
  name: bleble-svc
spec: 
  ports: 
     -  port: 8080
        protocol: TCP
        targetPort: 8000
  selector: 
    app: bleble
  type: NodePort

My Load balancer:

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      targetPort: http

My ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: simple-fanout-example
  namespace : default
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: internal-lb.aws.com
    http:
      paths:
      - path: /bleble
        backend:
          serviceName: bleble-svc
          servicePort: 80
      - path: /hello-world
        backend:
          serviceName: hello-world-svc
          servicePort: 80

I've set up the Nginx Ingress Controller with this : kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.24.1/deploy/mandatory.yaml

I am unsure why I get a 503 Service Temporarily Unavailable for one service and one 502 for another... I would guess it's a problem of ports or of namespace? In the guide, they don't define namespace for the deployment...

Every resources create correctly, and I think the ingress is actually working but is getting confused where to go.

Thanks for your help!

Answer 1

In general, use externalTrafficPolicy: Cluster instead of Local. You can gain some performance (latency) improvement by using Local but you need to configure those pod allocations with a lot efforts. You will hit 5xx errors with those misconfigurations. In addition, Cluster is the default option for externalTrafficPolicy.

In your ingress, you route /bleble to service bleble, but your service name is actually bleble-svc. please make them consistent. Also, you would need to set your servicePort to 8080 as you exposed 8080 in your service configuration.

For internal service like bleble-svc, Cluster IP is good enough in your case as it does not need external access.

Hope this helps.

Answer 2

Found it! The containerPort in the Deployment were set to 8000, the targetport of the services as well, but the person who did the Dockerfile of the code exposed the port 80. Which was the reason it was getting the 502 Bad getaway!

Thanks a lot as well to @Fei who has been a fantastic helper!