nginx errors readv() and recv() failed

Question

I use nginx along with fastcgi. I see a lot of the following errors in the error logs

readv() failed (104: Connection reset by peer) while reading upstream and recv() failed (104: Connection reset by peer) while reading response header from upstream

I don't see any problem using the application. Are these errors serious or how to get rid of them.

Answer 1

I was using php-fpm in the background and slow scripts were getting killed after a said timeout because it was configured that way. Thus, scripts taking longer than a specified time would get killed and nginx would report a recv or readv error as the connection is closed from the php-fpm engine/process.

Answer 2

Update:

Since nginx version 1.15.3 you can fix this by setting the keepalive_requests option of your upstream to the same number as your php-fpm's pm.max_requests:

upstream name {
    ...
    keepalive_requests number;
    ...
}

Original answer:

If you are using nginx to connect to php-fpm, one possible cause can also be having nginx' fastcgi_keep_conn parameter set to on (especially if you have a low pm.max_requests setting in php-fpm):

http|server|location {
    ...
    fastcgi_keep_conn on;
    ...
}

This may cause the described error every time a child process of php-fpm restarts (due to pm.max_requests being reached) while nginx is still connected to it. To test this, set pm.max_requests to a really low number (like 1) and see if you get even more of the above errors.

The fix is quite simple - just deactivate fastcgi_keep_conn:

fastcgi_keep_conn off;

Or remove the parameter completely (since the default value is off). This does mean your nginx will reconnect to php-fpm on every request, but the performance impact is negligible if you have both nginx and php-fpm on the same machine and connect via unix socket.

Answer 3

Regarding this error:

readv() failed (104: Connection reset by peer) while reading upstream and recv() failed (104: Connection reset by peer) while reading response header from upstream

there was 1 more case where I could still see this. Quick set up overview:

• CentOS 5.5
• PHP with PHP-FPM 5.3.8 (compiled from scratch with some 3rd party modules)
• Nginx 1.0.5

After looking at the PHP-FPM error logs as well and enabling catch_workers_output = yes in the php-fpm pool config, I found the root cause in this case was actually the amfext module (PHP module for Flash). There's a known bug and fix for this module that can be corrected by altering the amf.c file.

After fixing this PHP extension issue, the error above was no longer an issue.

Answer 4

This is a very vague error as it can mean a few things. The key is to look at all possible logs and figure it out. In my case, which is probably somewhat unique, I had a working nginx + php / fastcgi config. I wanted to compile a new updated version of PHP with PHP-FPM and I did so. The reason was that I was working on a live server that couldn't afford downtime. So I had to upgrade and move to PHP-FPM as seamlessly as possible.

Therefore I had 2 instances of PHP.

• 1 directly talking with fastcgi (PHP 5.3.4) - using TCP / 127.0.0.1:9000 (PHP 5.3.4)
• 1 configured with PHP-FPM - using Unix socket - unix:/dir/to/socket-fpm (PHP 5.3.8)

Once I started up PHP-FPM (PHP 5.3.8) on an nginx vhost using a socket connection instead of TCP I started getting this upstream error on any fastcgi page taking longer than x minutes whether they were using FPM or not. Typically it was pages doing large SELECTS in mysql that took ~2 min to load. Bad I know, but this is because of back end DB design.

What I did to fix it was add this in my vhost configuration: fastcgi_read_timeout 5m; Now this can be added in the nginx global fastcgi settings as well. It depends on your set up. http://wiki.nginx.org/HttpFcgiModule

Answer 5

Answer # 2. Interestingly enough fastcgi_read_timeout 5m; fixed one vhost for me. However I was still getting the error in another vhost, just by running phpinfo(); What fixed this for me was by copying over a default production php.ini file and adding the config I needed into it. What I had was an old copy of my php.ini from the previous PHP install. Once I put the default php.ini from 'shared' and just added in the extensions and config I needed, this solved my problem and no longer did I have nginx errors readv() and recv() failed.

I hope 1 of these 2 fixes helps someone.

Answer 6

Also it can be a very simple problem - there is an infinity cicle somewhere in your code, or an infinity trying to connect an external host on your page.