nginx 431 Request Header Fields Too Large

Question

Recently, the size of data in our cookies has become larger, and all of the requests that go through nginx started getting rejected with 431 error responses.

I tried increasing the large_client_header_buffers and client_header_buffer_size to no avail. Here is a sample of the main nginx.conf I am using:

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;
    large_client_header_buffers 4 64k;
    client_header_buffer_size 8k;

    include /etc/nginx/conf.d/*.conf;
}

And for the specific server block:

server {
    listen 443 ssl;

    server_name staging1.acme.services;

    ssl_certificate /certs/acme.services/fullchain.pem;
    ssl_certificate_key /certs/acme.services/privkey.pem;

    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 5m;

    client_max_body_size 1000M;

    # following two lines needed for DNS propagation to work
    resolver 127.0.0.11 ipv6=off valid=10s;
    set $v5 http://frontend;

    location /v1/ {
        proxy_pass http://gateway:4000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location / {
        proxy_pass $v5;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}

I have tried/looked at:

400 Bad Request Request Header Or Cookie Too Large nginx

"Request Header Or Cookie Too Large" in nginx with proxy_pass

Removing this nginx proxy and directly accessing the site removes these errors, so I'm pretty sure it's some configuration being made by this proxy. Here is the tcpdump produced by tcpdump -n -S -s 0 -A 'tcp dst port 80' | grep -B3 -A10 "GET"

17:41:49.205476 IP 172.23.0.2.46958 > 172.23.0.3.80: Flags [.], ack 1031354459, win 229, options [nop,nop,TS val 3490060934 ecr 2370799301], length 0
E..4M.@[email protected]..=y8[....XZ.....
.....O..
17:41:49.205513 IP 172.23.0.2.46958 > 172.23.0.3.80: Flags [P.], seq 71625227:71629509, ack 1031354459, win 229, options [nop,nop,TS val 3490060934 ecr 2370799301], length 4282: HTTP: GET / HTTP/1.0
E...M.@[email protected]..=y8[....i......
.....O..GET / HTTP/1.0
Host: staging.acme.services
X-Real-IP: 10.11.13.216
X-Forwarded-For: 10.11.13.216
X-Forwarded-Proto: https
Connection: close
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36
DNT: 1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
--
E..4.A@.@.;O.........p.P....$.......XZ.....
.....O.

Answer 1

Before going to backend, you add 3 more headers.

That could be enough to overflow the backend buffer, nearing its limits. You could try to reproduce the request using curl and passing all headers.

Just adjust backend.