How to set up a custom HTTP error in Kubernetes

Question

I want to create a custom 403 error page. Currently I already have an Ingress created and in the annotations I have something like this:

"nginx.ingress.kubernetes.io/whitelist-source-range": "100.01.128.0/20,88.100.01.01"

So any attempt to access my web app outside that IP range receives a 403 error.

In order to create a custom page I tried adding the following annotations:

"nginx.ingress.kubernetes.io/custom-http-errors": "403",
"nginx.ingress.kubernetes.io/default-backend": "default-http-backend"

where default-http-backend is the name of an app already deployed.

the ingress has this:

{
  "kind": "Ingress",
  "apiVersion": "extensions/v1beta1",
  "metadata": {
    "name": "my-app-ingress",
    "namespace": "my-app-test",
    "selfLink": "/apis/extensions/v1beta1/namespaces/my-app-test/ingresses/my-app-ingress",
    "uid": "8f31f2b4-428d-11ea-b15a-ee0dcf00d5a8",
    "resourceVersion": "129105581",
    "generation": 3,
    "creationTimestamp": "2020-01-29T11:50:34Z",
    "annotations": {
      "kubernetes.io/ingress.class": "nginx",
      "nginx.ingress.kubernetes.io/custom-http-errors": "403",
      "nginx.ingress.kubernetes.io/default-backend": "default-http-backend",
      "nginx.ingress.kubernetes.io/rewrite-target": "/",
      "nginx.ingress.kubernetes.io/whitelist-source-range": "100.01.128.0/20,90.108.01.012"
    }
  },
  "spec": {
    "tls": [
      {
        "hosts": [
          "my-app-test.retail-azure.js-devops.co.uk"
        ],
        "secretName": "ssl-secret"
      }
    ],
    "rules": [
      {
        "host": "my-app-test.retail-azure.js-devops.co.uk",
        "http": {
          "paths": [
            {
              "path": "/api",
              "backend": {
                "serviceName": "my-app-backend",
                "servicePort": 80
              }
            },
            {
              "path": "/",
              "backend": {
                "serviceName": "my-app-frontend",
                "servicePort": 80
              }
            }
          ]
        }
      }
    ]
  },
  "status": {
    "loadBalancer": {
      "ingress": [
        {}
      ]
    }
  }
}

Yet I always get the default 403. What am I missing?

Answer 1

I've reproduced your scenario and that worked for me. I will try to guide you in steps I've followed.

Cloud provider: GKE Kubernetes Version: v1.15.3 Namespace: default

I'm using 2 deployments of 2 images with a service for each one.

Service 1: default-http-backend - with nginx image, it will be our default backend.

Service 2: custom-http-backend - with inanimate/echo-server image, this service will be displayed if the request become from a whitelisted ip.

Ingress: Nginx ingress with annotations.

Expected behavior: The ingress will be configured to use default-backend, custom-http-errors and whitelist-source-range annotations. If the request was made from a whitelisted ip the ingress will redirect to custom-http-backend, if not it will be redirect to default-http-backend.

Deployment 1: default-http-backend

Create a file default-http-backend.yaml with this content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: default-http-backend
spec:
  selector:
    matchLabels:
      app: default-http-backend
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      containers:
      - name: default-http-backend
        image: nginx
        ports:
        - name: http
          containerPort: 80
        imagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
spec:
  selector:
    app: default-http-backend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

Apply the yaml file: k apply -f default-http-backend.yaml

Deployment 2: custom-http-backend

Create a file custom-http-backend.yaml with this content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-http-backend
spec:
  selector:
    matchLabels:
      app: custom-http-backend
  template:
    metadata:
      labels:
        app: custom-http-backend
    spec:
      containers:
      - name: custom-http-backend
        image: inanimate/echo-server
        ports:
        - name: http
          containerPort: 8080
        imagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
  name: custom-http-backend
spec:
  selector:
    app: custom-http-backend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

Apply the yaml file: k apply -f custom-http-backend.yaml

Check if services is up and running

I'm using the alias k for kubectl

➜  ~ k get svc                                 
NAME                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
custom-http-backend    ClusterIP   10.125.5.227   <none>        80/TCP    73s
default-http-backend   ClusterIP   10.125.9.218   <none>        80/TCP    5m41s
...

➜  ~ k get pods
NAME                                    READY   STATUS    RESTARTS   AGE
custom-http-backend-67844fb65d-k2mwl    1/1     Running   0          2m10s
default-http-backend-5485f569bd-fkd6f   1/1     Running   0          6m39s
...

You could test the service using port-forward:

default-http-backend k port-forward svc/default-http-backend 8080:80 Try to access http://localhost:8080 in your browse to see the nginx default page.

custom-http-backend k port-forward svc/custom-http-backend 8080:80 Try to access http://localhost:8080 in your browse to see the custom page provided by the echo-server image.

Ingress configuration

At this point we have both services up and running, we need to install and configure the nginx ingress. You can follow the official documentation, this will not covered here.

After installed let's deploy the ingress, based in the code you posted i did some modifications: tls removed, added other domain and removed the path /api for tests purposes only and add my home ip to whitelist.

Create a file my-app-ingress.yaml with the content:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-app-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: "/"
    nginx.ingress.kubernetes.io/custom-http-errors: '403'
    nginx.ingress.kubernetes.io/default-backend: default-http-backend
    nginx.ingress.kubernetes.io/whitelist-source-range: 207.34.xxx.xx/32
spec:
  rules:
  - host: myapp.rabello.me
    http:
      paths:
      - path: "/"
        backend:
          serviceName: custom-http-backend
          servicePort: 80

Apply the spec: k apply -f my-app-ingress.yaml

Check the ingress with the command:

➜  ~ k get ing
NAME             HOSTS              ADDRESS          PORTS   AGE
my-app-ingress   myapp.rabello.me   146.148.xx.xxx   80      36m

That's all!

If I test from home with my whitelisted ip, the custom page is showed, but if i try to access using my cellphone in 4G network, the nginx default page is displayed.

Note I'm using ingress and services in the same namespace, if you need work with different namespace you need to use ExternalName.

I hope that helps!

References: kubernetes deployments

kubernetes service

nginx ingress

nginx annotations

Answer 2

I want to create a custom 403 error page. Currently I already have an Ingress created and in the annotations. So any attempt to access my web app outside that IP range receives a 403 error. In order to create a custom page I tried adding the following annotations:

kind: Ingress
metadata:
  name: my-app-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: "/"
    nginx.ingress.kubernetes.io/custom-http-errors: '403'
    nginx.ingress.kubernetes.io/default-backend: default-http-backend
    nginx.ingress.kubernetes.io/whitelist-source-range: 125.10.156.36/32
spec:
  rules:
  - host: venkat.dev.vboffice.com
    http:
      paths:
      - path: "/"
        backend:
          serviceName: custom-http-backend
          servicePort: 80

where default-http-backend is the name of an app already deployed with default nginx page.

If I test from home with my whitelisted ip, the custom page is showed, but if i try to access using my cellphone in 4G network, it will display default backend 404

i need to add any nginx config change custom-http-backend pod????

Deployment 1:default-http-backend

apiVersion: apps/v1
kind: Deployment
metadata:
  name: default-http-backend
spec:
  selector:
    matchLabels:
      app: default-http-backend
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      containers:
      - name: default-http-backend
        image: nginx
        ports:
        - name: http
          containerPort: 80
        imagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
spec:
  selector:
    app: default-http-backend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

Deployment 2: custom-http-backend

apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-http-backend
spec:
  selector:
    matchLabels:
      app: custom-http-backend
  template:
    metadata:
      labels:
        app: custom-http-backend
    spec:
      containers:
      - name: custom-http-backend
        image: inanimate/echo-server
        ports:
        - name: http
          containerPort: 8080
        imagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
  name: custom-http-backend
spec:
  selector:
    app: custom-http-backend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080