How to debug Kubernetes nginx Ingress redirection from HTTP to HTTPS

Question

Fast question: how I can debug ingress and Nginx to know where exactly HTTP->HTTPS redirection happens?

More details:

What we have: we have war file + Tomcat, build it with Docker. Run it with Kubernetes in AWS.

What we need: the application should be accessible with HTTP and with HTTPS. HTTP should not redirect to HTTPS.

Problem: HTTP always redirects to HTTPS.

What we try: we have Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ${some name
  namespace: ${some namespace}
  labels:
    app: ${some app}
    env: ${some env}
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false" #we added this for turn off https redirection
    nginx.ingress.kubernetes.io/force-ssl-redirect: "false" #we added this for turn off https redirection
    nginx.ingress.kubernetes.io/affinity: "cookie" # We use it for sticky sessions
    nginx.ingress.kubernetes.io/affinity-mode: "persistent"
    nginx.ingress.kubernetes.io/session-cookie-name: "some cookie name"
    nginx.ingress.kubernetes.io/session-cookie-expires: "172800"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "172800"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/whitelist-source-range: ${whitelist of ip adresses}

spec:
  tls:
    - hosts:
        - ${some host}
        - ${some another host}
      secretName: my-ingress-ssl
  rules:
    - host: ${some host}
      http:
        paths:
          - path: /
            backend:
              serviceName: ${some another service name}
              servicePort: 8080
    - host: ${some another host}
      http:
        paths:
          - path: /
            backend:
              serviceName: ${some another service name}
              servicePort: 8080

And configmap

kind: ConfigMap
apiVersion: v1
metadata:
  labels:
    app: ${some app}
    env: ${some env}
  namespace: ${some namespace}
  name: nginx-config
data:
  hsts: "false" #we added this for turn off https redirection
  hsts-max-age: "0" #we added this for turn off https redirection
  ssl-redirect: "false" #we added this for turn off https redirection
  hsts-include-subdomains: "false" #we added this for turn off https redirection

In Tomcat server.xml we have:

<Connector port="8080" protocol="HTTP/1.1"
                   connectionTimeout="20000"
                   redirectPort="8443" />

...

<!-- Define an AJP 1.3 Connector on port 8009 -->
        <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

...

and this connector we commented (it shouldn't work now):

<!--
<Connector port="8443" protocol="org.apache.coyote.http11.Http11AprProtocol"
           maxThreads="150" SSLEnabled="true" >
    <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
    <SSLHostConfig>
        <Certificate certificateKeyFile="conf/key.pem"
                     certificateFile="conf/cert.pem"
                     certificateChainFile="conf/chain.pem"
                     type="RSA" />
    </SSLHostConfig>
</Connector>
-->

I tried all possible variants with ingress annotations, but without success result.

What I want to know: how can I debug ingress with Nginx to know where exactly HTTP->HTTPS redirection happens?

UPD

As it turned out, it was not the Nginx controller that was installed on the server, but Traefik. Due to security restrictions, I cant see that pod with the controller. So no Nginx annotations worked. Nevertheless, the answers below to my questions will help people with a similar problem.

Answer 1

In the setup you describe there are likley to be at least three processes involved in the handling of incoming http requests

internet -> AWS ELB -> Nginx Ingress Controller -> Tomcat

Any of the processes may be able to issue a 301 redirect to instruct http traffic to retry with https. A redirect in the tomcat process could either be specified by tomcat configuration or the app tomcat hosts.

I would attempt to isolate the process that is performing the redirect by experimenting to bypass the higher processes:

1. Bypass the loadbalancer. SSH onto one of the K8s nodes and curl the ingress services' nodeport in the URL (check out Internal Endpoints on the nginx-controller service description for the right port). If you get a redirect, you know the problem is not the the loadbalancer.
2. Bypass the ingress controllers by 'docker exec' into one of the app containers, then curl localhost:8080. If you get a redirect, you know the problem is not the the ingress controller, and must be in the tomcat config or application.

If none of this is possible due to ssh restrictions, curl availability etc... A different approach would be to turn on request logging for each of the processes involved - though this will not be appropriate for a production workload as it will lead to confidential information being output to inappropriate contexts, such as a web tier log file or cloudwatch logs.

Answer 2

1. You can have a look at nginx config for current nginx-ingress manifest & check for any rule for http to https redirection there, as described here, using a like below:
   kubectl exec -it -n <namespace-of-ingress-controller> <nginx-ingress-controller-pod-name> -- cat /etc/nginx/nginx.conf

2. Use command curl -LIk http://<domain> and check the server: header for the redirection. In curl:

   • -L will follow redirections
   • -I show only headers
   • -k will allow insecure SSL requests