Why does my bare-metal kubernetes nginx Ingress-controller return a 308?

Question

I am trying to create add the nginx ingress controller to my kubernetes cluster. My current cluster has 3 nodes and they all have open firewall rules in between them. (Note: This is a bare metal cluster for educational purposes)

I have created a deployment with the following config

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nodejs-test
  labels:
    app: nodejs
spec:
  replicas: 5
  selector:
    matchLabels:
      app: nodejs
  template:
    metadata:
      labels:
        app: nodejs
    spec:
      containers:
      - name: nodejs-container
        image: gcr.io/infrastructure/test-nodejs-server
        ports:
        - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: test-deployment-service
spec:
  selector:
    app: nodejs
  ports:
  - protocol: TCP
    port: 80
    targetPort: 3000

After running kubectl create -f file.yaml

I see all pods and services running. Within the cluster I can curl that service ip and get a response from the pods.

Now I go to create a ingress.

I first followed the install steps found at https://github.com/kubernetes/ingress-nginx/tree/master/deploy

I can see my ingress controller running.

    NAMESPACE       NAME                                         READY     STATUS    RESTARTS   AGE
default         my-test-6445d9bb7b-blm7t                     1/1       Running   0          3h
default         nodejs-test-5657f7bb74-7hwqk                 1/1       Running   0          23h
default         nodejs-test-5657f7bb74-bh44x                 1/1       Running   0          23h
default         nodejs-test-5657f7bb74-dw4g4                 1/1       Running   0          23h
default         nodejs-test-5657f7bb74-k7bf4                 1/1       Running   0          23h
default         nodejs-test-5657f7bb74-xdthl                 1/1       Running   0          23h
ingress-nginx   default-http-backend-55c6c69b88-4d5bf        1/1       Running   0          26m
ingress-nginx   nginx-ingress-controller-5c6698dfbf-gs9zd    1/1       Running   0          25m
kube-system     etcd-yoda.me.com                      1/1       Running   1          1d
kube-system     kube-apiserver-yoda.me.com            1/1       Running   1          1d
kube-system     kube-controller-manager-yoda.me.com   1/1       Running   1          1d
kube-system     kube-dns-6f4fd4bdf-62bxg                     3/3       Running   3          1d
kube-system     kube-flannel-ds-jbmwg                        1/1       Running   1          1d
kube-system     kube-flannel-ds-r8nfb                        1/1       Running   2          1d
kube-system     kube-flannel-ds-rczbx                        1/1       Running   0          1d
kube-system     kube-proxy-brm2g                             1/1       Running   0          1d
kube-system     kube-proxy-qjgcv                             1/1       Running   1          1d
kube-system     kube-proxy-sz6rw                             1/1       Running   1          1d
kube-system     kube-scheduler-yoda.me.com            1/1       Running   1          1d
kube-system     kubernetes-dashboard-5bd6f767c7-stscl        1/1       Running   1          23h
kube-system     tiller-deploy-59d854595c-tngg6               1/1       Running   0          23h

I then create my ingress with the following file

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    ingress.kubernetes.io/rewrite-target: /
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - http:
      paths:
      - path: /whoop
        backend:
          serviceName: test-deployment-service
          servicePort: 80

Then I run kubectl get services --all-namespaces to find my ingress controller port.

NAMESPACE       NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
default         kubernetes                ClusterIP   10.96.0.1        <none>        443/TCP                      1d
default         my-test                   ClusterIP   10.110.102.5     <none>        80/TCP                       4h
default         test-deployment-service   ClusterIP   10.104.239.37    <none>        80/TCP                       23h
ingress-nginx   default-http-backend      ClusterIP   10.105.218.87    <none>        80/TCP                       29m
ingress-nginx   ingress-nginx             NodePort    10.110.39.104    <none>        80:30652/TCP,443:31624/TCP   27m
kube-system     kube-dns                  ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP                1d
kube-system     kubernetes-dashboard      ClusterIP   10.100.219.110   <none>        443/TCP                      23h
kube-system     tiller-deploy             ClusterIP   10.101.179.202   <none>        44134/TCP                    23h

Now I go to curl my server:

curl -v myserver:30652/whoop/

The resulting response is:

*   Trying myip...
* TCP_NODELAY set
* Connected to myserver (myip) port 30652 (#0)
> GET /whoop/ HTTP/1.1
> Host: myserver:30652
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 308 Permanent Redirect
< Server: nginx/1.13.9
< Date: Fri, 02 Mar 2018 14:38:21 GMT
< Content-Type: text/html
< Content-Length: 187
< Connection: keep-alive
< Location: https://myserver:30652/whoop/
< Strict-Transport-Security: max-age=15724800; includeSubDomains;
< 
<html>
<head><title>308 Permanent Redirect</title></head>
<body bgcolor="white">
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx/1.13.9</center>
</body>
</html>
* Connection #0 to host myserver left intact

I would expect to be getting the response from my pods. However I always get the 308.

If I curl a random path

curl myserver:30652/random

The response I get is

default backend - 404

How can I hook up my ingress and ingress controller for the routing to work correctly?

Answer 1

The key here is to name a host, otherwise the ingress rule applies to the default server, which is configured to redirect HTTP to HTTPS by default.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    ingress.kubernetes.io/rewrite-target: /
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - path: /whoop
        backend:
          serviceName: test-deployment-service
          servicePort: 80

You should be able to disable the redirect behavior for the default server, bit I haven't tried that.

Answer 2

I was having the same problem, but I did not want to add the host value. To prevent the redirect HTTP to HTTPS by the default server like David mentioned, instead I added this annotation to the ingress and it fixed it for me: nginx.ingress.kubernetes.io/ssl-redirect: "false".

My working ingress looks like this now:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    kubernetes.io/ingress.class: "internal-nginx"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - http:
      paths:
      - path: /*
        backend:
          serviceName: app1
          servicePort: 80
      - path: /app2/*
        backend:
          serviceName: app2
          servicePort: 80

Answer 3

I was having issues with HTTP calls getting redirected to HTTPS (verified it with fiddler logs).

Adding this to my yml file made it work for me.

Annotations:

nginx.ingress.kubernetes.io/ssl-redirect: "false"