Nginx 502 Bad Gateway. Solved by increasing buffer. Why?

Question

I'm in the process of setting up a LEMP stack to run Drupal. I installed Nginx and PHP-FastCGI.

Nginx worked fine but any attempts to run PHP gave me the error "502 Bad Gateway".

A quick Google revealed: nginx 502 bad gateway, and increasing the buffer size solved the problem.

fastcgi_buffers 8 16k;
fastcgi_buffer_size 32k;

The question is why?

My understanding

From the previous link, it would seem that nginx was sending requests to PHP-FastCGI and it wasn't responding. What about these requests made it time out?

Did it not have enough time to respond because the php was complex (it wasn't, it was phpinfo();). Now I've increased the buffer, when should I worry about having to increase the buffer again?

Answer 1

If you will check nginx error log, most probably you will see this message:
upstream sent too big header while reading response header from upstream

fastcgi_buffers sets the number and memory size of buffer segments used for the response of the FastCGI upstream.

Default values, presented in documentation:
fastcgi_buffers 8 4k|8k;
where default buffer size is equal to the PAGESIZE of your operating system.
getconf PAGESIZE allows to get current memory page size.

For example, in Ubuntu 14.01, default PAGESIZE is 4KB. That means, you have 8 segments, 4KB each. Total is 32KB. Response of FastCGI is more than this number, that is the reason, why we get response code 502 - server received

It is not great explanation, but it will help you to understand better, I hope.

Answer 2

Actually, the problem is directly related only to fastcgi_buffer_size. This is a very special buffer that holds only the HTTP headers from response.

If your application emits a lot of Set-Cookie headers (or something else contributing to the total size of HTTP headers), the default buffer size here may not be sufficient and you need to increase it.

To understand how you need to increase it, you can read my super detailed writeup here - it's about proxy_buffer_size but fastcgi_ buffers behave very similarly. To quote the essential command:

curl -s -w \%{size_header} -o /dev/null https://example.com

Ensure to test against proper URL and add request headers via -H, if needed.

This will give you the header size in bytes. You will then need to align resulting value to 4k (typical size of memory page).

So if you got, e.g. 14342 bytes, then it's required to set:

fastcgi_buffer_size 16k;

The tricky part is not there, but rather in the fact that when you increase this buffer size, you need to increase either fastcgi_buffer_size and/or fastcgi_busy_buffers_size as well due to the way NGINX uses/calculates the default value for the latter.

Either way, don't set those buffers too high and use calculations specific to your app. Arbitrarily high values won't do good to your RAM, because those buffers are used per connection.

Answer 3

The problem is actually likely related to permissions in the container (we encountered this on Alpine but is likely the case on other distros too) on the /var/lib/nginx/tmp directory. That directory is owned by the nginx user and is writable only by the nginx group. When the request buffer exceeds the buffer size /var/lib/nginx/tmp is used as a temporary write location until enough of the buffer frees up to complete the request. The request to write to the tmp directory is made by the www-user(again in Alpine Linux) which doesn't have permissions to write to that location.

If you look in the pre-install script here for Nginx (again for Alpine Linux) you will find that the nginx group is being added to the www-data group. This is required for install as the nginx user is responsible for installing and booting the Nginx instance. After that all Nginx responsibilities are handed over to the www-data user which handles http traffic going through the container. In order for the www-data user to be able to write to the /var/lib/nginx/tmp directory, the ownership of the directory either needs to be changed to the www-data user or the www-data user needs to be added to the nginx group (which may bring up security concerns).

I have created an issue on the Nginx repo that explains this a little better and also contains a workaround if you are using Alpine Linux: https://gitlab.alpinelinux.org/alpine/aports/-/issues/12669

Although this is an Alpine Linux issue I suspect that others that are having issues with this are experiencing similar permissions issues. The Nginx docs explaining how this works can be found here.

I know this issue is old, but we recently ran into this and spent about a week figuring it out as simply increasing the buffer size didn't seem like a viable long-term solution for us. Hopefully this saves someone else from a week of headache trying to come across this same solution.