Running image with aws ecs throws 504 Gateway Time-out

Question

I dockerized my Application. If i run it with docker run, evertything works fine. I tried to run it with ecs fargate and put an ALB infront of it. If i try to access my Application via the ALB dns, i get an 504 Gateway Teme-out back.

While searching a solution, i found an post, which told me to set the Tomcat timeout higher than the ELB timeout, but it doesn't helped.

Dockerfile

FROM tomcat:8.0.20-jre8
RUN sed -i 's/connectionTimeout="20000"/connectionTimeout="70000"/' /usr/local/tomcat/conf/server.xml
CMD ["catalina.sh","run"]
COPY /target/Webshop.war /usr/local/tomcat/webapps/

ELB Log

http 2019-09-11T11:20:50.585293Z app/Doces-Backe-19RQJLVNHYG2P/8fb4f4079bb6ff9f 66.85.6.136:47767 - -1 -1 -1 503 - 18 348 "GET http://:8080/ HTTP/1.0" "-" - - arn:aws:elasticloadbalancing:eu-central-1:573575081005:targetgroup/ecs-Docest-de-webshop/8df4f0978484f8bd "Root=1-5d78d892-58886d3490906f0fa3914563" "-" "-" 0 2019-09-11T11:20:50.462000Z "forward" "-" "-"
http 2019-09-11T11:23:23.535869Z app/Doces-Backe-19RQJLVNHYG2P/8fb4f4079bb6ff9f 66.85.6.136:50950 10.10.11.140:8080 -1 -1 -1 504 - 18 303 "GET http://:8080/ HTTP/1.0" "-" - - arn:aws:elasticloadbalancing:eu-central-1:573575081005:targetgroup/ecs-Docest-de-webshop/8df4f0978484f8bd "Root=1-5d78d921-a236121716bd1bd209625fd8" "-" "-" 0 2019-09-11T11:23:13.415000Z "forward" "-" "-"
http 2019-09-11T11:23:56.286426Z app/Doces-Backe-19RQJLVNHYG2P/8fb4f4079bb6ff9f 66.85.6.136:51658 10.10.11.140:8080 -1 -1 -1 504 - 18 303 "GET http://:8080/ HTTP/1.0" "-" - - arn:aws:elasticloadbalancing:eu-central-1:573575081005:targetgroup/ecs-Docest-de-webshop/8df4f0978484f8bd "Root=1-5d78d942-22a1680464884762e02ec940" "-" "-" 0 2019-09-11T11:23:46.156000Z "forward" "-" "-"
http 2019-09-11T11:23:27.513803Z app/Doces-Backe-19RQJLVNHYG2P/8fb4f4079bb6ff9f 66.85.6.136:51034 10.10.11.140:8080 -1 -1 -1 504 - 18 303 "GET http://:8080/ HTTP/1.0" "-" - - arn:aws:elasticloadbalancing:eu-central-1:573575081005:targetgroup/ecs-Docest-de-webshop/8df4f0978484f8bd "Root=1-5d78d925-b6b5daf0d0f733140aea0f84" "-" "-" 0 2019-09-11T11:23:17.393000Z "forward" "-" "-"

I expected to see my application running at the elb.

Thanks for your help!

Answer 1

Solution:

The problem was that I set the correct port in the security group of the load balancer, but not in that of the ECS service. So I opened the required port there and now it works.

Procedure:

1. Go to your cluster
2. Go to the service with the problem
3. Click on the Security Group under the item Network Access and open the required port

Thanks!

Answer 2

There can be multiple reasons behind gateway timeout. The only thing that I do not like about fargate is debug-log. @AWS team should enable log configuration for fargate service by default as its hard to debug these issues without logs.

Better to configure log driver and push logs to cloud watch and see the actual issue also double check your desired port in task definition and mapped port in service.

            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "awslogs-spring",
                    "awslogs-region": "us-west-2",
                    "awslogs-stream-prefix": "awslogs-example"
                }

or from AWS console

You need to assign permission or role of cloud watch logs to task definition or service to push the logs to Cloud watch.

Once logs are configured then goto cloudwatch loggroup and search the log group so you will insight to your application.

But still, to troubleshoot the actual issue first, you have to understand the error code and possible reason of Gateway Timeout.

HTTP 504: Gateway Timeout

Description: Indicates that the load balancer closed a connection because a request did not complete within the idle timeout period.

Cause 1: The application takes longer to respond than the configured idle timeout.

Solution 1: Monitor the HTTPCode_ELB_5XX and Latency metrics. If there is an increase in these metrics, it could be due to the application not responding within the idle timeout period. For details about the requests that are timing out, enable access logs on the load balancer and review the 504 response codes in the logs that are generated by Elastic Load Balancing. If necessary, you can increase your capacity or increase the configured idle timeout so that lengthy operations (such as uploading a large file) can complete. For more information, see Configure the Idle Connection Timeout for Your Classic Load Balancer and How do I troubleshoot Elastic Load Balancing high latency.

Cause 2: Registered instances closing the connection to Elastic Load Balancing.

Solution 2: Enable keep-alive settings on your EC2 instances and make sure that the keep-alive timeout is greater than the idle timeout settings of your load balancer.