Nginx HTTPS->HTTP gets 403 "SSL required" error with Spring

Question

The problem

I'm getting 403 SSL required from Spring when trying to route through my ELB to Kubernetes Nginx ingress controller.

Setup

My set up is as follows:

1. I've got an ELB (AWS) with ACM for my Kubernetes cluster (created by kops) which routes all requests to the
2. Nginx Ingress Controller which in turn routes all requests according to the rules dictated in the
3. Ingress that passes the traffic unto the
4. Service that exposes port 80 and routes in to port 8080 in the
5. Pods selected with labels "app=foobar" (which are described in a Deployment)
6. Pods are running a Spring Boot Web App v2.1.3

So basically: https://foo.bar.com(:443) -> ingress -> http://foo.bar.svc.cluster.local:80

This works like a charm for everything. Except SprintBoot. For some reason, I keep getting 403 - SSL required from Spring

One note to keep in mind here: my Spring application does not have anything to do with SSL. I don't want it to do anything in that nature. For this example's purposes, this should be a regular REST API requests, with the SSL termination happening outside the container.

What I tried so far

• Port-forwarding to the service itself and requesting - it works fine.
• Disabling CSRF in WebSecurityConfigurerAdapter
• Putting ingress annotation nginx.ingress.kubernetes.io/force-ssl-redirect=true - it gives out TOO_MANY_REDIRECTS error when I try it (instead of the 403)
• Putting ingress annotation nginx.ingress.kubernetes.io/ssl-redirect=true - doesn't do anything
• Putting ingress annotation nginx.ingress.kubernetes.io/enable-cors: "true" - doesn't do anything
• Also nginx.ingress.kubernetes.io/ssl-passthrough: "true"
• Also nginx.ingress.kubernetes.io/secure-backends: "true"
• Also kubernetes.io/tls-acme: "true"
• I tried a whole bunch of other stuff that I can't really remember right now

How it all looks like in my cluster

Nginx ingress controller annotations look like this (I'm using the official nginx ingress controller helm chart, with very little modifications other than this thing):

service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "aws_acm_certificate_arn"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"

Ingress looks like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: foobar
  namespace: api
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - backend:
          serviceName: foobar
          servicePort: http
        path: /

Service looks like this:

apiVersion: v1
kind: Service
metadata:
  name: foobar
  namespace: api
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: foobar

What I think the problem is

My hunch is that it's something with X-Forwarded headers, and Spring doing its magic behind the scenes, trying to be all smart like and deciding that I need SSL based on some headers without me explicitly asking for it. But I didn't figure it out yet.

---

I searched far and wide for a solution, but I couldn't find anything to ease my pain... hope you'll be able to help!

Edit

I found out that my current setup (without k8s and nginx) works fine and ELB passes X-Forwarded-Port: 443 and X-Forwarded-Proto: https, and it seems to work, but on my k8s cluster with nginx, I put in a listener client that spits out all the headers, and my headers seem to be X-Forwarded-Port: 80 and X-Forwarded-Proto: http

Answer 1

Thanks for all the people that helped out, I actually found the answer. Within the code there were some validations that all requests should come from a secure source, and Nginx Ingress Controller changed these headers (X-Forwarded-Proto and X-Forwarded-Port) because SSL was terminated within ELB and handed to the ingress controller as HTTP

To fix that I did the following:

1. Added use-proxy-protocol: true to the config map - which passed the correct headers, but got errors regarding broken connection (which I don't really remember the actual error right now, I'll edit this answer later if there will be any requests for it)

2. To fix these errors I added the following the the nginx ingress controller annotations configuration:

service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"

This made sure that all traffic will use the proxy protocol, and I also had to change the backend-protocol from HTTP to TCP.

Doing this made sure all requests routed ELB reserve their original X headers, and are passed unto Nginx Ingress Controller, which in turn passed unto my applications that require these headers to be passed.