GKE Internal Load Balancer is failing to create

Question

I have a gke cluster that has an http(s) load balancer, which uses the RATE balancing mode as per the docs.

I'm trying to expose a workload on this cluster to App Engine using an internal load balancer. Both services are on the same network in the same region.

However, when I try to create the load balancer, it fails with this error.

Error creating load balancer (will retry): failed to ensure load balancer for service default/internal-es-lb: googleapi: Error 400: Validation failed for instance 'projects/PROJECT-NAME/zones/us-central1-a/instances/gke-staging-default-pool-85830c52-g6tg': instance may belong to at most one load-balanced instance group., instanceInMultipleLoadBalancedIgs

There are a couple of things that are weird.
1. The internal LB worked earlier today.
2. The http(s) load balancer is (was) throwing the same error as the internal LB, even when it's the only load balancer on the cluster
3. When I create the LB with my service yaml, it creates a new/different instance group than my target pool

Here's my service yaml:

apiVersion: v1
kind: Service
metadata:
  name: internal-es-lb
  annotations:
    cloud.google.com/load-balancer-type: "Internal"
  labels:
    app: internal-es-lb
spec:
  type: LoadBalancer
  loadBalancerIP: 10.128.0.4
  loadBalancerSourceRanges: [0.0.0.0/0]
  ports:
    - port: 80
      targetPort: 9200
      protocol: TCP
      name: http-es-lb
  selector:
    app: elastic-master

I think I understand the error to mean that the there can only be one instance group, so is it possible to specify the instance group in the yaml? Or, is there some other solution? Thanks!

Update: the internal LBs were working earlier today because I hadn't implemented the http(s) LB yet. With the http(s) lb in place, these fail. Would love to know how to make this all work together since the docs aren't very clear.

Update to the Update: If I create the internal LBs first and then the http(s) LB applied to the new instance group, it sorta works. Everything is being routed and loaded correctly, but the google console throws a bunch of errors, so I don't know if this is the recommended way.

Answer 1

When you create a HTTP(S) LB, you are likely creating a backend that uses a Managed Instance Group (MIG). Instances (including GKE Nodes) can only be part of a single MIG at any given time.

When you create a ILB service through GKE, the backends are all unmanaged instance groups, so the instances can be reused.

The recommended way to address this is to use Kubernetes Ingress instead of the GCE L7LB since the Ingress will also use unmanaged instance groups.