Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

AWS Network ELB take 4 minutes to recognise target as healthy

Using AWS Network ELBs: takes at least four minutes for a registered instance to become 'healthy'. The instances and services have been running for days, I am simply de-registering and then registering on the same target group, as part of a deployment. If I use a script or use the AWS UI, CLI it makes no difference.

Healthcheck Settings are:

  • Port: tried various, all have listening services tested via curl. 80,22,9001
  • Healthy threshold: 2
  • Unhealthy threshold: 2
  • Timeout: 10
  • Interval: 30

I can see the connection requests coming in on whatever port has been specified, the service responds appropriately and the connection is then closed. As far as I am aware this should be sufficient for the ELB to determine that the instance is healthy (once the threshold has been passed). Which should mean that my instances are up and running no more than 90 seconds past registration time. I have no idea why this should be happening, should be straight forward.

I cannot determine what would be causing such a long delay given that I have fulfilled the known criteria for my instances being healthy. They sit at the Elb.InitialHealthChecking reason for about 4 minutes. Any ideas on further tests to determine the cause of the delay?

like image 477
Tim Lindsay Avatar asked Nov 13 '17 01:11

Tim Lindsay


People also ask

How does NLB do health check?

Network Load Balancers use active and passive health checks to determine whether a target is available to handle requests. By default, each load balancer node routes requests only to the healthy targets in its Availability Zone.

What response code shows the ELB that an instance is healthy?

An instance is considered healthy if it returns a 200 response code within the health check interval.

Why is ELB health check failing?

Your registered instances can fail the load balancer health check for several reasons. The most common reasons for failing a health check are where EC2 instances close connections to your load balancer or where the response from the EC2 instances times out.


2 Answers

We hit this issue with NLBs and raised it with AWS support on March 20, 2018. Their response:

This is a known issue, where a newly registered instance remains in initial state for longer period of time and our internal team is already working on the fix for this issue. Unfortunately, at this point in time we do not have an ETA for the fix.

They confirmed that, under normal circumstances, targets should remain in the initial state until HealthyThreshold health checks pass.

like image 84
Matthew Riley Avatar answered Oct 21 '22 05:10

Matthew Riley


I can confirm this doesn't work till September 2021. I thought I did something wrong, but, everything was okay and the process to healthy status after 4-5 minutes. :|

enter image description here

like image 41
Arefe Avatar answered Oct 21 '22 03:10

Arefe