Kubernetes Ingress: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io"

Question

Playing around with K8 and ingress in local minikube setup. Creating ingress from yaml file in networking.k8s.io/v1 api version fails. See below output. Executing

> kubectl apply -f ingress.yaml

returns

Error from server (InternalError): error when creating "ingress.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": an error on the server ("") has prevented the request from succeeding

in local minikube environment with hyperkit as vm driver.

Here is the ingress.yaml file:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mongodb-express-ingress
  namespace: hello-world
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
    - host: mongodb-express.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: mongodb-express-service-internal
                port:
                  number: 8081

Here is the mongodb-express deployment file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-express
  namespace: hello-world
  labels:
    app: mongodb-express
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongodb-express
  template:
    metadata:
      labels:
        app: mongodb-express
    spec:
      containers:
        - name: mongodb-express
          image: mongo-express
          ports:
            - containerPort: 8081
          env:
            - name: ME_CONFIG_MONGODB_ADMINUSERNAME
              valueFrom:
                secretKeyRef:
                  name: mongodb-secret
                  key: mongodb-root-username
            - name: ME_CONFIG_MONGODB_ADMINPASSWORD
              valueFrom:
                secretKeyRef:
                  name: mongodb-secret
                  key: mongodb-root-password
            - name: ME_CONFIG_MONGODB_SERVER
              valueFrom:
                configMapKeyRef:
                  name: mongodb-configmap
                  key: mongodb_url
---
apiVersion: v1
kind: Service
metadata:
  name: mongodb-express-service-external
  namespace: hello-world
spec:
  selector:
    app: mongodb-express
  type: LoadBalancer
  ports:
    - protocol: TCP
      port: 8081
      targetPort: 8081
      nodePort: 30000
---
apiVersion: v1
kind: Service
metadata:
  name: mongodb-express-service-internal
  namespace: hello-world
spec:
  selector:
    app: mongodb-express
  ports:
    - protocol: TCP
      port: 8081
      targetPort: 8081

Some more information:

> kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

> minikube version
minikube version: v1.19.0
commit: 15cede53bdc5fe242228853e737333b09d4336b5

> kubectl get all -n hello-world
NAME                                   READY   STATUS    RESTARTS   AGE
pod/mongodb-68d675ddd7-p4fh7           1/1     Running   0          3h29m
pod/mongodb-express-6586846c4c-5nfg7   1/1     Running   6          3h29m

NAME                                       TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
service/mongodb-express-service-external   LoadBalancer   10.106.185.132   <pending>     8081:30000/TCP   3h29m
service/mongodb-express-service-internal   ClusterIP      10.103.122.120   <none>        8081/TCP         3h3m
service/mongodb-service                    ClusterIP      10.96.197.136    <none>        27017/TCP        3h29m

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mongodb           1/1     1            1           3h29m
deployment.apps/mongodb-express   1/1     1            1           3h29m

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/mongodb-68d675ddd7           1         1         1       3h29m
replicaset.apps/mongodb-express-6586846c4c   1         1         1       3h29m

> minikube addons enable ingress
    ▪ Using image k8s.gcr.io/ingress-nginx/controller:v0.44.0
    ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
    ▪ Using image docker.io/jettech/kube-webhook-certgen:v1.5.1
🔎  Verifying ingress addon...
🌟  The 'ingress' addon is enabled

> kubectl get all -n ingress-nginx
NAME                                            READY   STATUS      RESTARTS   AGE
pod/ingress-nginx-admission-create-2bn8h        0/1     Completed   0          4h4m
pod/ingress-nginx-admission-patch-vsdqn         0/1     Completed   0          4h4m
pod/ingress-nginx-controller-5d88495688-n6f67   1/1     Running     0          4h4m

NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/ingress-nginx-controller             NodePort    10.111.176.223   <none>        80:32740/TCP,443:30636/TCP   4h4m
service/ingress-nginx-controller-admission   ClusterIP   10.97.107.77     <none>        443/TCP                      4h4m

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-nginx-controller   1/1     1            1           4h4m

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/ingress-nginx-controller-5d88495688   1         1         1       4h4m

NAME                                       COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   1/1           7s         4h4m
job.batch/ingress-nginx-admission-patch    1/1           9s         4h4m

---

However, it works for the beta api version, i.e.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: mongodb-express-ingress-deprecated
  namespace: hello-world
spec:
  rules:
    - host: mongodb-express.local
      http:
        paths:
          - path: /
            backend:
              serviceName: mongodb-express-service-internal
              servicePort: 8081

Any help very much appreciated.

Answer 1

I had the same issue. I successfully fixed it using:

kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission

then apply the yaml files:

kubectl apply -f ingress_file.yaml

Answer 2

I have the same problem with you, and you can see this issue https://github.com/kubernetes/minikube/issues/11121.

Two way you can try:

1. download the new version ,or go back the old version
2. Do a strange thing like what balnbibarbi said.

2. The Strange Thing

# Run without --addons=ingress
sudo minikube start --vm-driver=none #--addons=ingress

# install external ingress-nginx
sudo helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
sudo helm repo update
sudo helm install ingress-nginx ingress-nginx/ingress-nginx

# expose your services

And then you will find your Ingress lacks Endpoints. And then:

sudo minikube addons enable ingress

After minitues, the Endpoints appears.

Problem

If you search examples with addons Ingress by Google, you will find what the below lacks is ingress.

root@ubuntu:~# kubectl get pods -n kube-system 
NAME                             READY   STATUS    RESTARTS   AGE
coredns-74ff55c5b-xnmx2          1/1     Running   1          4h40m
etcd-ubuntu                      1/1     Running   1          4h40m
kube-apiserver-ubuntu            1/1     Running   1          4h40m
kube-controller-manager-ubuntu   1/1     Running   1          4h40m
kube-proxy-k9lnl                 1/1     Running   1          4h40m
kube-scheduler-ubuntu            1/1     Running   2          4h40m
storage-provisioner              1/1     Running   3          4h40m

Answer 3

Ref: Expecting apiVersion - networking.k8s.io/v1 instead of extensions/v1beta1

TL;DR

kubectl explain predated a lot of the generic resource parsing logic, so it has a dedicated --api-version flag. This should do what you want.

kubectl explain ingresses --api-version=networking.k8s.io/v1

This should solve your doubt!