Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

AWS Load Balancer EC2 health check request timed out failure

I'm trying to get down and dirty with DevOps and I'm running into a health check request timed out failure. The problem is my Elastic Load Balancer sends a health check to my EC2 instance and gets a network timeout. I'm not sure what I did wrong. I am following this tutorial and I have completed all the steps up to and including "Using a Elastic Load Balancer". My EC2 instance seems to be working fine and I am able to successfully curl localhost on port 9292 from within the EC2 instance.

EC2 instance security group setup: description

Elastic Load Balancer setup: enter image description here

My target group for the ELB routing has port 9292 open via HTTP and here's a screenshot of the target in my target group that is unhealthy. enter image description here

Health check config:enter image description here

I have a VPC that my EC2 instance is a part of and my ELB is connected to the same VPC. I do not have Apache installed and I do not have nginx installed. To my understanding, I do not need these. I have a Rails Puma server running and I can send successful curl requests to the server.

My hunch is that my ELB is not allowed to reach my EC2 instance, resulting in a network timeout and a failed health check. I'm unable to find the cause for this. Any ideas? This SO post didn't help much. Are my security groups misconfigured? What else could potentially block a routing request from ELB to my EC2 instance?

Also, is there a way to view network requests / logs for my EC2 instance? I keep seeing VPC flow logging but I feel like there are simpler alternatives.

Here's something I posted in the AWS forums but to no avail.

UPDATE: I can curl the private IP of target just fine from within an EC2 instance. I don't think it's the target instance, I think it's something to do with the security group setup. I am unable to identify why though because I have basically allowed all traffic from the Load Balancer to the EC2 instance.

like image 935
Sticky Avatar asked Jun 07 '17 05:06

Sticky


People also ask

What happens when an EC2 instance behind an ELB fails a health check?

If an instance fails these status checks, it is marked unhealthy and is terminated while Amazon EC2 Auto Scaling launches a new replacement instance. You can attach one or more load balancer target groups, one or more Classic Load Balancers, or both to your Auto Scaling group.

What ELB will do if one of the instance failed health check?

Health check failed The instance will continue being monitored and if it starts failing health checks, the ELB will respond by marking it as unhealthy, stop routing traffic to it, and wait for the ASG to replace it.

Why is ELB health check failing?

An instance might fail the ELB health check because an application running on the instance has issues that cause the load balancer to consider the instance out of service.

What is EC2 health check and ELB health check?

EC2 instance health check. Elastic Load Balancer (ELB) health check. Auto Scaling and Custom health checks. Amazon EC2 performs automated checks on every running EC2 instance to identify hardware and software issues. Status checks are performed every minute and each returns a pass or a fail status.


1 Answers

I made my mistake during the "Setup your VPC" step. I finished creating a subnet for an RDS instance. I proceeded to start an instance and the default subnet that AWS chose when I switched to my VPC was the subnet I made for my RDS, which was NOT a public subnet. Therefore, any attempts, from any EC2 instance or my load balancer, would not be able to reach it because I had only set up my public subnet to take requests.

The solution was to create a new instance and this time, pick the correct public subnet. My original EC2 instance was associated with a private subnet while the load balancer was pointing to the public subnet.

Here's a link to a hand drawn image that helped me pin point my problem, hopefully can help anyone else who's having trouble setting up. I didn't put image here directly because it's bigger than 2MB.

Glad to answer any further questions too!

like image 150
Sticky Avatar answered Sep 25 '22 12:09

Sticky