AWS Elastic Beanstalk 504 Gateway Timeout

Question

I have a Node server running on AWS Elastic Beanstalk. One of my endpoints accepts a huge payload and the function itself is pretty slow and lengthy, and can take upwards of 10+ minutes.

Due to business requirements it must remain as a single HTTP POST and cannot be split up to be any smaller.

On larger calls I am getting a 504 GATEWAY TIMEOUT, always around the 60 second mark. I have tried toying with the timeout settings in Elastic Beanstalk Load Balancer section to no avail, it seems the longest timeout duration is 60 second anyway.

I did see one promising solution in the docs at https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/config-idle-timeout.html

To ensure that lengthy operations such as file uploads have time to complete, send at least 1 byte of data before each idle timeout period elapses

This sounds like exactly what I need, but I do not know how to accomplish

1) How can I "send at least 1 byte of data" from my Node app to ensure the session is kept alive and doesn't timeout after one minutes

Answer 1

504 Timeouts on ElasticBeanstalk environments can be resolved by settings your ELB policies. Updating Nginx timeout configurations may also be required.

• ELB policies: Set the Idle Timeout of your Elastic Load Balancer to the value of your choice (defaults to 60s). To do this, create a .ebextensions folder in the root of your project. In this folder create another file with a .config file extension, and set the ELB Idle Timeout to the value of your choice (e.g. 300 seconds):

   option_settings:
     - namespace: aws:elb:policies
       option_name: ConnectionSettingIdleTimeout
       value: 300
  

  Or if you are using an application load balancer:

   option_settings:
     - namespace: aws:elbv2:loadbalancer
       option_name: IdleTimeout
       value: 300
  

• Nginx configuration: Set Nginx the with the desired value of timeout: send_timeout, proxy_connect_timeout, proxy_read_timeout, proxy_Send_timeout all default to 60s (additional specification to check may be: client_header_timeout, client_body_timeout, keepalive_timeout). Default timeout values in Nginx are specified in the specifications and configurations files (e.g. the .config files under /etc/nginx). In the .ebextensions folder create a new file (e.g. 01-nginx-configuration.config or update in the .config file above), and append the following content inside the file (add or remove relevant settings according to the timeouts observed):

   files:
     "/etc/nginx/conf.d/nginx.custom.conf":
         mode: "644"
         owner: "root"
         group: "root"
         content: |
           client_header_timeout   300;
           client_body_timeout     300;
           send_timeout            300;
           proxy_connect_timeout   300;
           proxy_read_timeout      300;
           proxy_send_timeout      300;
  
   container_commands:
     01_restart_nginx:
       command: "sudo service nginx reload"
  

There are several more ways to add this configuration. Read more here and here.

Edit (06/11/2020): Some people indicated that the keep_alive 300; configuration crushed their server. Therefore, I've removed that from the nginx.custom.conf configuration.

Answer 2

In order to update the nginx config on the Amazon Linux 2 platform I successfully managed to increase the timeout limits by extending the nginx config following these instructions: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/platforms-linux-extend.html

/.platform/nginx/conf.d/myconf.conf:

client_header_timeout   300;
client_body_timeout     300;
send_timeout            300;
proxy_connect_timeout   300;
proxy_read_timeout      300;
proxy_send_timeout      300;

Answer 3

In my case error was happening because there was not enough memory for application container.

Increasing memory in containerDefinitions in Dockerrun.aws.json file solved my problem easily

{
  "AWSEBDockerrunVersion": 2,
  "volumes": [],
  "containerDefinitions": [
    {
      "name": "###",
      "image": "###",
      "essential": true,
      "memory": 521,         # formerly 128
...