502 Bad Gateway: nginx, php5-fpm, 175/209 connect() failed (111: Connection refused) while connecting to upstream

Question

Running shopware 5 on a Debian Jessie machine with nginx and php5-fpm, we get very often a 502 Bad Gateway. This happens mostly in backend when longer operations are working like thumbnail creation, even if this is done within small chunks of single ajax requests.

The used server with 64 GB RAM and 16 Cores is sleeping at all, because there is no real traffic on it. We use it like a staging system currently unless we have fixed all errors like this one.

Error log:

In the nginx-error log the following lines can be found then:

[error] 20524#0: *175 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xx.xx.xx, server: domain.com, request: "POST /backend/MediaManager/createThumbnails HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.domain.com", referrer: "http://www.domain.com/backend/"

[error] 20524#0: *175 no live upstreams while connecting to upstream, client: xx.xx.xx.xx, server: domain.com, request: "POST /backend/Log/createLog HTTP/1.1", upstream: "fastcgi://php-fpm", host: "www.domain.com", referrer: "http://www.domain.com/backend/"

[error] 20524#0: *175 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xx.xx.xx, server: domain.com, request: "GET /backend/login/getLoginStatus?_dc=1457014588680 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.domain.com", referrer: "http://www.domain.com/backend/"

[error] 20522#0: *209 connect() failed (111: Connection refused) while connecting to upstream, client: xx.xx.xx.xx, server: domain.com, request: "GET /backend/login/getLoginStatus?_dc=1457014618682 HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.domain.com", referrer: "http://www.domain.com/backend/"

Maybe it is notable, that at first lot of "*175 connect" errors occure and then finally a "*209 connect".

Config files:

I'll try to post only significant lines related to this topic and will leave out all those lines which are commented out.

php-fpm:

/etc/php5-fpm/pool.d/www.conf:

[www]
user = www-data
group = www-data
listen = /var/run/php5-fpm.sock
listen.owner = www-data
listen.group = www-data
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3

nginx:

/etc/nginx/nginx.conf:

user www-data;
worker_processes auto;
pid /run/nginx.pid;

events {
    worker_connections 768;
    multi_accept on;
}

http {
    ## MIME types.
    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ## Default log and error files.
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    ## Use sendfile() syscall to speed up I/O operations and speed up
    ## static file serving.
    sendfile        on;

    ## Handling of IPs in proxied and load balancing situations.
    # set_real_ip_from        192.168.1.0/24; # set to your proxies ip or range
    # real_ip_header          X-Forwarded-For;

    ## Timeouts.
    client_body_timeout             60;
    client_header_timeout           60;
    keepalive_timeout            10 10;
    send_timeout                    60;

    ## Reset lingering timed out connections. Deflect DDoS.
    reset_timedout_connection on;

    ## Body size.
    client_max_body_size 10m;

    ## TCP options.
    tcp_nodelay        on;
    ## Optimization of socket handling when using sendfile.
    tcp_nopush         on;

    ## Compression.
    gzip              on;
    gzip_buffers      16 8k;
    gzip_comp_level   1;
    gzip_http_version 1.1;
    gzip_min_length   10;
    gzip_types        text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon application/vnd.ms-fontobject font/opentype application/x-font-ttf;
    gzip_vary         on;
    gzip_proxied      any; # Compression for all requests.
    gzip_disable      "msie6";

    ## Hide the Nginx version number.
    server_tokens off;

    ## Upstream to abstract backend connection(s) for PHP.
    upstream php-fpm {
        server unix:/var/run/php5-fpm.sock;
        # server 127.0.0.1:9000;

        ## Create a backend connection cache.
        keepalive 32;
    }

    ## Include additional configs
    include /etc/nginx/conf.d/*.conf;

    ## Include all vhosts.
    include /etc/nginx/sites-enabled/*;
}

/etc/nginx/sites-available/site.conf:

server {
    listen 80;
    listen 443 ssl;

    server_name xxxxxxxx.com;
    root /var/www/shopware;

    ## Access and error logs.
    access_log /var/log/nginx/xxxxxxxx.com.access.log;
    error_log  /var/log/nginx/xxxxxxxx.com.error.log;

    ## leaving out lots of shopware/mediafiles-related settings
    ## ....
    ## continue:

    location ~ \.php$ {
        try_files $uri $uri/ =404;

        ## NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini
        fastcgi_split_path_info ^(.+\.php)(/.+)$;

        ## required for upstream keepalive
        # disabled due to failed connections
        #fastcgi_keep_conn on;

        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param SHOPWARE_ENV    $shopware_env if_not_empty;
        fastcgi_param ENV             $shopware_env if_not_empty; # BC for older SW versions

        fastcgi_buffers 8 16k;
        fastcgi_buffer_size 32k;

        client_max_body_size 24M;
        client_body_buffer_size 128k;

        ## upstream "php-fpm" must be configured in http context
        fastcgi_pass php-fpm;
    }

}

What to do now? Please let me now if i should provide further information to this question.

Update

After applying nginx- and fpm-settings from @peixotorms, the errors in nginx-logs changed to:

30 upstream timed out (110: Connection timed out) while reading response header from upstream

But the issue itself isn't solved. It has just another face...

Answer 1

It might sound strange to you, but your problem is most probably due to the fact that you're running PHP on a socket instead of a tcp port. You will start seeing 502 errors (and others) when you have around 300 concurrent requests (sometimes less) to php on a socket configuration.

Also your pm.max_children is way too low, unless you want to limit your server to around 5 simultaneous php requests maximum: http://php.net/manual/en/install.fpm.configuration.php

Configure it this way, and those errors should go away:

For your nginx.conf change the following values:

worker_processes 4; 
worker_rlimit_nofile 750000;

# handles connection stuff
events { 
worker_connections 50000;
multi_accept on;
use epoll;
}

upstream php-fpm {
        keepalive 30;
        server 127.0.0.1:9001;
}

Your /etc/php5-fpm/pool.d/www.conf

(Use these settings because you have plenty or RAM and CPU)

[www]
user = www-data
group = www-data
listen.owner = www-data
listen.group = www-data
listen.mode = 0660
listen = 127.0.0.1:9001
listen.allowed_clients = 127.0.0.1
listen.backlog = 65000
pm = dynamic
pm.max_children = 1024
pm.start_servers = 8
pm.min_spare_servers = 4
pm.max_spare_servers = 16
pm.max_requests = 10000

Also add this on your location ~ \.php$ { block:

location ~ \.php$ {
        try_files $uri $uri/ =404;

        ## NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini
        fastcgi_split_path_info ^(.+\.php)(/.+)$;

        ## required for upstream keepalive
        # disabled due to failed connections
        #fastcgi_keep_conn on;

        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param SHOPWARE_ENV    $shopware_env if_not_empty;
        fastcgi_param ENV             $shopware_env if_not_empty; # BC for older SW versions

        fastcgi_keep_conn on;
        fastcgi_connect_timeout 20s; 
        fastcgi_send_timeout 60s; 
        fastcgi_read_timeout 60s;
        fastcgi_pass php-fpm;
    }

EDIT:

Change the values below on your /etc/php5/fpm/php.ini file to this and restart:

safe_mode = Off    
output_buffering = Off
zlib.output_compression = Off
max_execution_time = 900
max_input_time = 900
memory_limit = 2048M
post_max_size = 120M
file_uploads = On
upload_max_filesize = 120M