GKE Ingress with container-native load balancing does not detect health check (Invalid value for field 'resource.httpHealthCheck')

Question

I am running a cluster on Google Kubernetes Engine and I am currently trying to switch from using an Ingress with external load balancing (and NodePort services) to an ingress with container-native load balancing (and ClusterIP services) following this documentation: Container native load balancing

To communicate with my services I am using the following ingress configuration that used to work just fine when using NodePort services instead of ClusterIP:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: mw-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: mw-cluster-ip
    networking.gke.io/managed-certificates: mw-certificate
    kubernetes.io/ingress.allow-http: "false"
spec:
  rules:
    - http:
        paths:
          - path: /*
            backend:
              serviceName: billing-frontend-service
              servicePort: 80
          - path: /auth/api/*
            backend:
              serviceName: auth-service
              servicePort: 8083

Now following the documentation, instead of using a readinessProbe as a part of the container deployment as a health check I switched to using ClusterIP services in combination with BackendConfig instead. For each deployment I am using a service like this:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: auth
  name: auth-service
  namespace: default
  annotations:
    cloud.google.com/backend-config: '{"default": "auth-hc-config"}'
spec:
  type: ClusterIP
  selector:
    app: auth
  ports:
    - port: 8083
      protocol: TCP
      targetPort: 8083

And a Backend config:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: auth-hc-config
spec:
  healthCheck:
    checkIntervalSec: 10
    port: 8083
    type: http
    requestPath: /auth/health

As a reference, this is what the readinessProbe used to look like before:

          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /auth/health
              port: 8083
              scheme: HTTP
            periodSeconds: 10

Now to the actual problem. I deploy the containers and services first and they seem to startup just fine. The ingress however does not seem to pick up the health checks properly and shows this in the Cloud console:

Error during sync: error running backend syncing routine: error ensuring health check: googleapi: Error 400: Invalid value for field 'resource.httpHealthCheck': ''. HTTP healthCheck missing., invalid

The cluster as well as the node pool are running GKE version 1.17.6-gke.11 so the annotation cloud.google.com/neg: '{"ingress": true}' is not necessary. I have checked and the service is annotated correctly:

Annotations:       cloud.google.com/backend-config: {"default": "auth-hc-config"}
                   cloud.google.com/neg: {"ingress":true}
                   cloud.google.com/neg-status: {"network_endpoint_groups":{"8083":"k8s1-2078beeb-default-auth-service-8083-16a14039"},"zones":["europe-west3-b"]}

I have already tried to re-create the cluster and the node-pool with no effect. Any ideas on how to resolve this? Am I missing an additional health check somewhere?

Answer 1

I found my issue. Apparently the BackendConfig's type attribute is case-sensitive. Once I changed it from http to HTTP it worked after I recreated the ingress.