Mysterious Http 408 errors in AWS elasticbeanstalk-access_log

Question

The elasticbeanstalk-access_log log-file in our AWS EBS instances are full of 408 errors, like these:

172.31.1.56 (-) - - [16/Mar/2016:10:16:31 +0000] "-" 408 - "-" "-"
172.31.1.56 (-) - - [16/Mar/2016:10:16:31 +0000] "-" 408 - "-" "-"
172.31.1.56 (-) - - [16/Mar/2016:10:16:31 +0000] "-" 408 - "-" "-"
172.31.1.56 (-) - - [16/Mar/2016:10:16:31 +0000] "-" 408 - "-" "-"
172.31.1.56 (-) - - [16/Mar/2016:10:16:31 +0000] "-" 408 - "-" "-"
172.31.1.56 (-) - - [16/Mar/2016:10:16:59 +0000] "-" 408 - "-" "-"

They appear randomly, sometimes there are a few minutes between them, sometimes there are 4-6 errors within a few seconds. These errors also happen on our non-public staging environment when there is no any real traffic on the server, so the source of these requests is probably one of AWS's own service.

Answer 1

EDIT: Do you have a classic load balancer? Change to the application load balancer by creating a new environment with the Elastic Beanstalk cli and select application load balancer. This will solve this issue.

---

ELB has a mechanism called pre-open connections. The ELB does this so that your requests can be served faster, i.e. your new requests will not have to wait at the ELB for the extra time it will take to open up a fresh connection to the backend before the requests get sent to the backends. If you have a lower keep-alive timeout, causing pre-opened connections to be closed quicker which will make your backend generate a 408 error response to indicate that the connection was closed because the client (ELB) timeout expired without the ELB sending any request on that particular connection.

If you have modified the ELB idle connection timeout, then you need to make sure your http keep-alive timeout value is greater than the ELB idle connection timeout value. If this is not the case enable keep alive timeouts and make sure the value is greater than the ELB idle connection timeout.

You can change the keepalive timeout in apache by adding a .config file in the ebextensions folder with the following code:

files:
  "/etc/httpd/conf.d/keepalive.conf" :
  mode: "000644"
  owner: root
  group: root
  content: |
    # Enable TCP keepalive
    Timeout 300
    KeepAlive On
    MaxKeepAliveRequests 100
    KeepAliveTimeout 300
    <IfModule mod_reqtimeout.c>
     RequestReadTimeout header=300 body=300
    </IfModule>