Nginx: broken_header with proxy_protocol and ELB

Question

I am trying to set up proxy_protocol in my nginx config. My server sits behind an AWS load balancer (ELB), and I have enabled Proxy Protocol on that for both ports 80 and 443.

However, this is what I get when I hit my server:

broken header: "��/��
                                                             '���\DW�Vc�A{����
                                                                              �@��kj98���=5���g@32ED�</A
    " while reading PROXY protocol, client: 172.31.12.223, server: 0.0.0.0:443

That is a direct copy paste from the nginx error log - wonky characters and all.

Here is a snip from my nginx config:

server {
  listen  80 proxy_protocol;
  set_real_ip_from 172.31.0.0/20; # Coming from ELB
  real_ip_header proxy_protocol;
  return  301 https://$http_host$request_uri;
}

server {
  listen      443 ssl proxy_protocol;
  server_name *.....com
  ssl_certificate      /etc/ssl/<....>;
  ssl_certificate_key  /etc/ssl/<....?;
  ssl_prefer_server_ciphers On;
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!DSS:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4;
  ssl_session_cache shared:SSL:10m;
  add_header X-Frame-Options DENY;
  add_header X-Content-Type-Options nosniff;
  ssl_stapling on;
  ssl_stapling_verify on;


  ...

I can't find any help online about this issue. Other people have had broken header issues, but the error with bad headers are always readable - they don't look like they are encoded like they are for me.

Any ideas?

Answer 1

Two suggestions:

1. Verify that your ELB listener is configured to use TCP as the protocol, not HTTP. I have an LB config like the following that's routing to Nginx with proxy_protocol configured:

   {
     "LoadBalancerName": "my-lb",
     "Listeners": [
        {
         "Protocol": "TCP",
         "LoadBalancerPort": 80,
         "InstanceProtocol": "TCP",
         "InstancePort": 80
       }
     ],
     "AvailabilityZones": [
       "us-east-1a",
       "us-east-1b",
       "us-east-1d",
       "us-east-1e"
     ],
     "SecurityGroups": [
        "sg-mysg"
     ]
   }
   

2. You mentioned that you have enabled Proxy Protocol in the ELB, so I'm assuming you've followed AWS setup steps. If so then the ELB should be crafting the HTTP request correctly with the first line as something like PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n. However if the HTTP request is not coming into Nginx with the PROXY ... line then it could cause the problem you're seeing. You could reproduce that if you hit the EC2 DNS name directly in the browser, or you ssh into the EC2 instance and try something like curl localhost, then you should see a similar broken header error in the Nginx logs.

To find out whether it works with a correctly formed HTTP request you can use telnet:

    $ telnet localhost 80
    PROXY TCP4 198.51.100.22 203.0.113.7 35646 80
    GET /index.html HTTP/1.1
    Host: your-nginx-config-server_name
    Connection: Keep-Alive

Then check the Nginx logs and see if you have the same broken header error. If not then the ELB is likely not sending the properly formatted PROXY request, and I'd suggest re-doing the ELB Proxy Protocol configuration, maybe with a new LB, to verify it's set up correctly.

Answer 2

I had similar situation, nginx had 'proxy_protocol' on but AWS ELB settings was not on, so I got the similar message.

Solutions to edit settings to turn it on:

Answer 3

I had this error and came across this ticket:

• https://trac.nginx.org/nginx/ticket/886

which ultimately led me to figuring out that I had an unneeded proxy_protocol declaration in my nginx.conf file. I removed that and everything was working again.

Oddly enough, everything worked fine with nginx version 1.8.0, but when I upgraded to nginx version 1.8.1 is when I started seeing the error.