Kubernetes nginx ingress periodically gives 404

Question

I have deployed kops k8s in AWS, everything in the same namespace.

nginx ingress controller route traffic to https backends (wordpress apps).

I'm able to reach the website, but unfortunately for every 10~ calls only 1 call get http 200. all the other 9 get 404 nginx not found. tried to search everywhere but no luck :(

My configuration: DNS -> AWS NLB -> 2 Nodes

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-nginx
  namespace: example-ns
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/ssl-passthrough: "True"
    nginx.org/ssl-services: test-service
    nginx.ingress.kubernetes.io/affinity: "cookie"
spec:
  rules:
  - host: "test.example.com"
    http:
      paths:
      - path: /
        backend:
          serviceName: test-service
          servicePort: 8443

nginx-service.yaml:

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: example-ns
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      targetPort: http
    - name: https
      port: 443
      targetPort: https

nginx-daemonset.yaml:

kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: nginx-ingress-controller
  namespace: example-ns
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      imagePullSecrets:
      - name: private-repo
      containers:
        - name: nginx-ingress-controller
          image: private_repo/private_image
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
            - --default-ssl-certificate=$(POD_NAMESPACE)/tls-cert
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            runAsUser: 33
          resources:
            limits:
              cpu: 500m
              memory: 300Mi
            requests:
              cpu: 400m
              memory: 200Mi
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: https
              containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10

wordpress.yaml:

apiVersion: apps/v1
kind: Deployment

metadata:
  name: test-example
  namespace: example-ns
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
    spec:
      restartPolicy: Always
      volumes:
      - name: volume
        persistentVolumeClaim:
           claimName: volume-claim
      imagePullSecrets:
      - name: private-repo

      containers:
      - name: test-example-httpd
        image: private_repo/private_image
        imagePullPolicy: Always
        ports:
        - containerPort: 8443
          name: https

      - name: test-example-php-fpm
        image: private_repo/private_image
        imagePullPolicy: Always
        securityContext:
          runAsUser: 82
        securityContext:
          allowPrivilegeEscalation: false

---
apiVersion: v1
kind: Service
metadata:
  name: test-service
  namespace: example-ns
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  ports:
  - name: https-web
    targetPort: 8443
    port: 8443
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---UPDATE---

kubectl get  endpoints,services -n example-ns
NAME                           ENDPOINTS                                            AGE
endpoints/ingress-nginx        100.101.0.1:8443,100.100.0.4:443,100.101.0.2:443     1d
endpoints/test-service   100.100.0.1:8443,100.101.0.1:8443,100.101.0.2:8443   4h

NAME                         TYPE           CLUSTER-IP       EXTERNAL-IP                                                                     PORT(S)                      AGE
service/ingress-nginx        LoadBalancer   SOME-IP     sometext.elb.us-west-3.amazonaws.com   80:31541/TCP,443:31017/TCP   1d
service/test-service   ClusterIP      SOME-IP   <none>                                                                          8443/TCP                     4h

Thanks!

Answer 1

Apparently changing the annotation nginx.ingress.kubernetes.io/ssl-passthrough from "True" to "False" solved it.

Probably has to do something with ssl termination in NGINX and not in the apache.