Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Mysterious Http 408 errors in AWS elasticbeanstalk-access_log

The elasticbeanstalk-access_log log-file in our AWS EBS instances are full of 408 errors, like these:

172.31.1.56 (-) - - [16/Mar/2016:10:16:31 +0000] "-" 408 - "-" "-"
172.31.1.56 (-) - - [16/Mar/2016:10:16:31 +0000] "-" 408 - "-" "-"
172.31.1.56 (-) - - [16/Mar/2016:10:16:31 +0000] "-" 408 - "-" "-"
172.31.1.56 (-) - - [16/Mar/2016:10:16:31 +0000] "-" 408 - "-" "-"
172.31.1.56 (-) - - [16/Mar/2016:10:16:31 +0000] "-" 408 - "-" "-"
172.31.1.56 (-) - - [16/Mar/2016:10:16:59 +0000] "-" 408 - "-" "-"

They appear randomly, sometimes there are a few minutes between them, sometimes there are 4-6 errors within a few seconds. These errors also happen on our non-public staging environment when there is no any real traffic on the server, so the source of these requests is probably one of AWS's own service.

like image 553
Selindek Avatar asked Mar 23 '16 09:03

Selindek


People also ask

Why load balancer is not working?

If the load balancer is not responding to requests, check for the following issues: Your internet-facing load balancer is attached to a private subnet. You must specify public subnets for your load balancer. A public subnet has a route to the Internet Gateway for your virtual private cloud (VPC).

What causes ELB 4XX?

ELB 4XX errors HTTP 4XX error codes are generated when clients send faulty or malformed requests to the load balancer. Though potential causes for these errors can be guessed, not much can be done to troubleshoot.

How do I enable keep alive settings on an EC2 instance?

For backend connections, we recommend that you enable the HTTP keep-alive option for your EC2 instances. You can enable HTTP keep-alive in the web server settings for your EC2 instances. If you enable HTTP keep-alive, the load balancer can reuse backend connections until the keep-alive timeout expires.


1 Answers

EDIT: Do you have a classic load balancer? Change to the application load balancer by creating a new environment with the Elastic Beanstalk cli and select application load balancer. This will solve this issue.


ELB has a mechanism called pre-open connections. The ELB does this so that your requests can be served faster, i.e. your new requests will not have to wait at the ELB for the extra time it will take to open up a fresh connection to the backend before the requests get sent to the backends. If you have a lower keep-alive timeout, causing pre-opened connections to be closed quicker which will make your backend generate a 408 error response to indicate that the connection was closed because the client (ELB) timeout expired without the ELB sending any request on that particular connection.

If you have modified the ELB idle connection timeout, then you need to make sure your http keep-alive timeout value is greater than the ELB idle connection timeout value. If this is not the case enable keep alive timeouts and make sure the value is greater than the ELB idle connection timeout.

You can change the keepalive timeout in apache by adding a .config file in the ebextensions folder with the following code:

files:
  "/etc/httpd/conf.d/keepalive.conf" :
  mode: "000644"
  owner: root
  group: root
  content: |
    # Enable TCP keepalive
    Timeout 300
    KeepAlive On
    MaxKeepAliveRequests 100
    KeepAliveTimeout 300
    <IfModule mod_reqtimeout.c>
     RequestReadTimeout header=300 body=300
    </IfModule>
like image 142
Feltkamp.tv Multimedia Avatar answered Nov 24 '22 00:11

Feltkamp.tv Multimedia