Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

AWS ECS error: Task failed ELB health checks in Target group

I am using cloud formation template to build the infrastructure (ECS fargate cluster). Template executed successfully and stack has been created successfully. However, task has failed with the following error:

Task failed ELB health checks in (target-group arn:aws:elasticloadbalancing:eu-central-1:890543041640:targetgroup/prc-service-devTargetGroup/97e3566c8b307abf) 

I am not getting what and where to look for this to troubleshoot the issue. as it is fargate cluster, I am not getting how to login to container and execute some health check queries to debug further.

Can someone please help me to guide further on this and help me? Due to this error, I am not even able to access my web app. As ALB won't route the traffic if it is unhealthy.

What I did

After some googling, I found this post: https://aws.amazon.com/premiumsupport/knowledge-center/troubleshoot-unhealthy-checks-ecs/

However, I guess, this is related to EC2 compatibility in fargate. But in my case, EC2 is not there.

If you feel, I can paste the entire template as well.

please help

like image 446
user2315104 Avatar asked Feb 03 '19 13:02

user2315104


People also ask

Why is ELB health check failing?

An instance might fail the ELB health check because an application running on the instance has issues that cause the load balancer to consider the instance out of service.

What ELB will do if one of the instance failed health check?

Health check failed The instance will continue being monitored and if it starts failing health checks, the ELB will respond by marking it as unhealthy, stop routing traffic to it, and wait for the ASG to replace it.


2 Answers

This is resolved. It was the issue with the following points:

  • Docker container port mapping with host port were incorrect
  • ALB health check interval time was very short. Due to that, ALB was giving up immediately, not waiting for docker container to up and running properly.

after making these changes, it worked properly

like image 185
user2315104 Avatar answered Oct 08 '22 14:10

user2315104


There are quite a few of different possible reasons for this issue, not only the open ports:

  • Improper IAM permissions for the ecsServiceRole IAM role
  • Container instance security group Elastic Load Balancing load
  • balancer not configured for all Availability Zones Elastic Load
  • Balancing load balancer health check misconfigured
  • Unable to update the service servicename: Load balancer container name or port changed in task definition

Therefore AWS created an own website in order to address the possibilities of this error:

https://docs.aws.amazon.com/en_en/AmazonECS/latest/developerguide/troubleshoot-service-load-balancers.html

Edit: in my case the health check code of my application was different. The default is 200 but you can also add a range such as 200-499.

like image 32
Rene B. Avatar answered Oct 08 '22 12:10

Rene B.