Load Balancer : Health Check fails for no reason

Question

I am trying to set up a load balancer on GCE for 2 Tomcat servers, running individually on 2 VMs (vm-1 and vm-2). Both listen on port 80 and the network firewall rules allow traffic on port 80 from any source (0.0.0.0/0). Now, I created an instance group of both these VMs together called vm-group and set a named port named http pointing to port 80.

I created a health check too, on port 80, pointing to /<app_name>/<health_url>, which reports a HTTP 200 if the app is healthy.

Then, I setup a HTTP load balancer using instructions in this video. Once setup, I find that the load balancer reports that 0/2 instances are healthy, which means the health checks are failing.

When I manually hit the health check URLs, they return a HTTP 200 - so my app is healthy.

Now, I am not sure why the load balancer reports the VMs as unhealthy and is unable to route requests. How can I debug this further?

Edit: I verified that the google-address-manager is running as mentioned in this question.

Answer 1

Have you added google's health checker to your firewall list: 130.211.0.0/22, 35.191.0.0/16

Answer 2

Make sure firewall rules are explicitly allowing tomcat listened port(s) both on the VM node and on the GCP firewall.

1. To explicitly open VM node ports via iptables or firewall-cmd on Redhat/Centos based distros.

2. To explicitly open GCP firewall ports, create ingress firewall rules with ports specified, pay attention to the rule assignments, in GCP, it's called "target" https://cloud.google.com/vpc/docs/firewalls#rule_assignment In my case, I forgot to assign the created firewall rules assignments, aka relate the target tags to the VM nodes, the firewall rules will not be in effect until they are assigned (associated the target tags with the target VM, see below). After fixing/associating the tags with the firewall rules, viola! everything works.