DNS and nginx server setup causes slow server and 502 response

Question

I'm setting up a new server with Ubuntu 18.04.1 Nginx 1.14.0 and PHP 7.2.7 Everything works fine except a test page where I set up a lot of broken links to missing images. Edit: I decided to ask a new question based on the conclusions of this discussion that there probably is a DNS misconfiguration.

It seems to take forever for the server to realise they are missing and respond to the http request. Some missing files give a HTTP status of 404 and some give 502. What causes these delays and 502 errors? Did I do something wrong in the nginx or php configuration?

On my old server I have the exact same page which loads very quick. There must be a difference in server setup that I would like to solve.

Edit: the server doesn't recognize the the IPv6 address being server1.eu but accessing the IPv4 address immediately shows https://server1.eu

The zone file records:

AAAA    server1.eu  directs to the IPv6 address 
A   server1.eu  directs to the IPv4 address           

Dayo suggested the hosts file might be a problem, however the problem persists if the hosts file is emptied all together. Here is the contents:

127.0.0.1 localhost
::1 localhost
(the IPv6 address) localhost
127.0.1.1 host.server1.eu


# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

The nginx server configuration, when I remove the line "listen [::]:443 ssl http2;" everything works fine:

server {
        listen 443 ssl http2;
        listen [::]:443 ssl http2;
        ssl_certificate /etc/letsencrypt/live/server1.eu/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/server1.eu/privkey.pem;
        include snippets/ssl-params.conf;

        server_name server1.eu;
        root /var/www/server1/webroot;
        index index.php index.html index.htm ;

        location / {
            try_files $uri $uri/ /index.php?$args;
        }

        location ~ \.php$ {
            include fastcgi.conf;
            fastcgi_pass unix:/run/php/php7.2-fpm.sock;
        }
}

nginx.conf

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
    worker_connections 2048;
    multi_accept on;
}

http {

    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    #   keepalive_timeout 65;
    types_hash_max_size 2048;
    # server_tokens off;

    # server_names_hash_bucket_size 64;
    # server_name_in_redirect off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # SSL Settings
    ##

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
    ssl_prefer_server_ciphers on;

    ##
    # Logging Settings
    ##

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    ##
    # Gzip Settings
    ##

    gzip             on;
    gzip_comp_level  2;
    gzip_min_length  1000;
    gzip_proxied     expired no-cache no-store private auth;
    gzip_types       text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

    ##
    # Virtual Host Configs
    ##

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
    client_body_buffer_size 10K;
    client_header_buffer_size 1k;
    client_max_body_size 100m;
    large_client_header_buffers 4 8k;
    fastcgi_buffers 16 16k;
    fastcgi_buffer_size 32k;
    fastcgi_read_timeout 500; #gateway probleem
    client_body_timeout 12;
    client_header_timeout 12;
    keepalive_timeout 25;
    send_timeout 10;
}

The php app I use is Joomla 3.8.11 with a custom script to show a custom error page:

header("HTTP/1.0 404 Not Found"); 
echo file_get_contents('https://server1.eu/404-page-not-found'); 
exit;

After removing file_get_contents there are no errors anymore. However I'm wondering why, as it used to work fine on my old server (see edit above about DNS). Also I need this script to properly show a HTTP status 404 and a custom error page without changing the addressbar.

A part of the nginx error.log:

2018/08/30 16:25:27 [error] 29228#29228: *76 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 2a02:a440:91e3:1:4481:654b:a3e8:9617, server: server1.eu, request: "GET /images/klanten1/JHoogeveen.gif HTTP/2.0", upstream: "fastcgi://unix:/run/php/php7.2-fpm.sock:", host: "server1.eu", referrer: "https://server1.eu/portfolio-2"

Messages from the php7.2-fpm.log (there are a lot similar lines)

[30-Aug-2018 16:16:08] WARNING: [pool www] server reached pm.max_children setting (15), consider raising it
[30-Aug-2018 16:16:27] WARNING: [pool www] child 29026, script '/var/www/server1/webroot/index.php' (request: "GET /index.php") execution timed out (22.937711 sec), terminating
[30-Aug-2018 16:16:27] WARNING: [pool www] child 29245 exited on signal 15 (SIGTERM) after 20.490546 seconds from start
[30-Aug-2018 16:16:27] NOTICE: [pool www] child 29263 started

Below is the timeline of HTTP requests and replies, that shows the requests on non-existent files and the response of the server, some give a 404 which is good, some give 502 bad gateway (on my old server they are all 404's). Edit: There are fifteen 502 responses, which number is the maximum amount of concurrent PHP processes as Miknik remarked in his answer.

GET https://server1.eu/portfolio-2 [HTTP/2.0 200 OK 132ms]
GET https://server1.eu/templates/purity_iii/css/bootstrap.css [HTTP/2.0 200 OK 40ms]
GET https://server1.eu/templates/system/css/system.css [HTTP/2.0 200 OK 50ms]
GET https://server1.eu/templates/purity_iii/css/template.css [HTTP/2.0 200 OK 50ms]
GET https://server1.eu/templates/purity_iii/fonts/font-awesome/css/font-awesome.min.css [HTTP/2.0 200 OK 50ms]
GET https://server1.eu/templates/purity_iii/css/layouts/corporate.css [HTTP/2.0 200 OK 50ms]
GET https://server1.eu/media/jui/js/jquery.min.js?48b6d1b3850bca834b403c58682b4579 [HTTP/2.0 200 OK 60ms]
GET https://server1.eu/media/jui/js/jquery-noconflict.js?48b6d1b3850bca834b403c58682b4579 [HTTP/2.0 200 OK 60ms]
GET https://server1.eu/media/jui/js/jquery-migrate.min.js?48b6d1b3850bca834b403c58682b4579 [HTTP/2.0 200 OK 60ms]
GET https://server1.eu/media/system/js/caption.js?48b6d1b3850bca834b403c58682b4579 [HTTP/2.0 200 OK 70ms]
GET https://server1.eu/plugins/system/t3/base-bs3/bootstrap/js/bootstrap.js? 8b6d1b3850bca834b403c58682b4579 [HTTP/2.0 200 OK 80ms]
GET https://server1.eu/plugins/system/t3/base-bs3/js/jquery.tap.min.js [HTTP/2.0 200 OK 80ms]
GET https://server1.eu/plugins/system/t3/base-bs3/js/script.js [HTTP/2.0 200 OK 70ms]
GET https://server1.eu/plugins/system/t3/base-bs3/js/menu.js [HTTP/2.0 200 OK 70ms]
GET https://server1.eu/templates/purity_iii/js/script.js [HTTP/2.0 200 OK 70ms]
GET https://server1.eu/plugins/system/t3/base-bs3/js/nav-collapse.js [HTTP/2.0 200 OK 70ms]
GET https://server1.eu/templates/purity_iii/css/custom-vuyk.css [HTTP/2.0 200 OK 70ms]
GET https://server1.eu/images/klanten1/schipper2.gif [HTTP/2.0 502 Bad Gateway 23988ms]
GET https://server1.eu/images/klanten1/Kuiper.gif [HTTP/2.0 502 Bad Gateway 24038ms]
GET https://server1.eu/images/klanten1/WindMatch.gif [HTTP/2.0 502 Bad Gateway 24008ms]
GET https://server1.eu/images/klanten1/Tuinland.gif [HTTP/2.0 502 Bad Gateway 24018ms]
GET https://server1.eu/images/klanten1/Wezenberg.gif [HTTP/2.0 502 Bad Gateway 24038ms]
GET https://server1.eu/images/klanten1/Morgenster.gif [HTTP/2.0 502 Bad Gateway 23998ms]
GET https://server1.eu/images/klanten1/Harrie-boerhof.gif [HTTP/2.0 502 Bad Gateway 24028ms]
GET https://server1.eu/images/klanten1/Lococensus.gif [HTTP/2.0 502 Bad Gateway 23998ms]
GET https://server1.eu/images/klanten1/JHoogeveen.gif [HTTP/2.0 502 Bad Gateway 23978ms]
GET https://server1.eu/images/klanten1/DeDeur.gif [HTTP/2.0 502 Bad Gateway 23988ms]
GET https://server1.eu/images/klanten1/Runhaar.gif [HTTP/2.0 502 Bad Gateway 23958ms]
GET https://server1.eu/images/klanten1/Schunselaar-schildersbedrijf.gif [HTTP/2.0 502 Bad Gateway 23948ms]
GET https://server1.eu/images/klanten1/Capelle.gif [HTTP/2.0 502 Bad Gateway 23958ms]
GET https://server1.eu/images/klanten1/Distantlake.gif [HTTP/2.0 502 Bad Gateway 24038ms]
GET https://server1.eu/images/klanten1/Eikenaar.gif [HTTP/2.0 502 Bad Gateway 24018ms]
GET https://server1.eu/images/klanten1/FFWD.gif [HTTP/2.0 404 Not Found 26274ms]
GET https://server1.eu/images/klanten1/Veltec.gif [HTTP/2.0 404 Not Found 26791ms]
GET https://server1.eu/images/klanten1/Heutink.gif [HTTP/2.0 404 Not Found 26811ms]
GET https://server1.eu/images/klanten1/Lindeboom.gif [HTTP/2.0 404 Not Found 26777ms]
GET https://server1.eu/images/klanten1/aataxi.gif [HTTP/2.0 404 Not Found 26828ms]
GET https://server1.eu/images/klanten1/Aewind.gif [HTTP/2.0 404 Not Found 26811ms]
GET https://server1.eu/images/klanten1/Praatmaatgroep.gif [HTTP/2.0 404 Not Found 26800ms]
GET https://server1.eu/media/system/css/system.css [HTTP/2.0 200 OK 20ms]
JQMIGRATE: Migrate is installed, version 1.4.1 jquery-migrate.min.js:2:542
GET https://server1.eu/images/logo.gif [HTTP/2.0 200 OK 20ms]
GET https://server1.eu/images/reclame-en-communicatie.gif [HTTP/2.0 200 OK 20ms]
GET https://server1.eu/fonts/opensans-regular-webfont.woff [HTTP/2.0 200 OK 40ms]
GET https://server1.eu/templates/purity_iii/fonts/font-awesome/fonts/fontawesome-webfont.woff2?v=4.7.0 [HTTP/2.0 200 OK 70ms]

fastcgi.conf

fastcgi_param  PATH_TRANSLATED    $document_root$fastcgi_path_info;
fastcgi_param  SCRIPT_FILENAME    $document_root$fastcgi_script_name;
fastcgi_param  PATH_INFO          $fastcgi_path_info;
fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;

fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;
fastcgi_param  REQUEST_SCHEME     $scheme;
fastcgi_param  HTTPS              $https if_not_empty;

fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;

fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;

# PHP only, required if PHP was built with --enable-force-cgi-redirect
fastcgi_param  REDIRECT_STATUS    200;

php.ini

[PHP]

engine = On
short_open_tag = Off
precision = 14
output_buffering = 4096
zlib.output_compression = Off
implicit_flush = Off
unserialize_callback_func =
serialize_precision = -1
disable_functions = pcntl_alarm,pcntl_fork,pcntl_waitpid,pcntl_wait,pcntl_wifexited,pcntl_wifstopped,pcntl_wifsignaled,pcntl_wifcontinued,pcntl_wexitstatus,pcntl_wtermsig,pcntl_wstopsig,pcntl_signal,pcntl_signal_get_handler,pcntl_signal_dispatch,pcntl_get_last_error,pcntl_strerror,pcntl_sigprocmask,pcntl_sigwaitinfo,pcntl_sigtimedwait,pcntl_exec,pcntl_getpriority,pcntl_setpriority,pcntl_async_signals,
disable_classes =
zend.enable_gc = On
expose_php = Off
max_execution_time = 30
max_input_time = 60
memory_limit = 128M
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
display_errors = Off
display_startup_errors = Off
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
html_errors = On
variables_order = "GPCS"
request_order = "GP"
register_argc_argv = Off
auto_globals_jit = On
post_max_size = 28M
auto_prepend_file =
auto_append_file =
default_mimetype = "text/html"
default_charset = "UTF-8"
doc_root =
user_dir =
enable_dl = Off
cgi.fix_pathinfo=1
file_uploads = On
upload_max_filesize = 20M
max_file_uploads = 20
allow_url_fopen = On
allow_url_include = Off
default_socket_timeout = 60
cli_server.color = On
date.timezone = "Europe/Amsterdam"

[Pdo_mysql]
pdo_mysql.cache_size = 2000
pdo_mysql.default_socket=
[mail function]
SMTP = localhost
smtp_port = 25
mail.add_x_header = Off

[ODBC]
odbc.allow_persistent = On
odbc.check_persistent = On
odbc.max_persistent = -1
odbc.max_links = -1
odbc.defaultlrl = 4096
odbc.defaultbinmode = 1

[Interbase]
ibase.allow_persistent = 1
ibase.max_persistent = -1
ibase.max_links = -1
ibase.timestampformat = "%Y-%m-%d %H:%M:%S"
ibase.dateformat = "%Y-%m-%d"
ibase.timeformat = "%H:%M:%S"

[MySQLi]
mysqli.max_persistent = -1
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.cache_size = 2000
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect = Off
[mysqlnd]
mysqlnd.collect_statistics = On
mysqlnd.collect_memory_statistics = Off

[PostgreSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0

[bcmath]
bcmath.scale = 0

[Session]
session.save_handler = files
session.use_strict_mode = 0
session.use_cookies = 1
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.serialize_handler = php
session.gc_probability = 0
session.gc_divisor = 1000
session.gc_maxlifetime = 1440
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.sid_length = 26
session.trans_sid_tags = "a=href,area=href,frame=src,form="
session.sid_bits_per_character = 5

[Assertion]
zend.assertions = -1

[mbstring]
mbstring.func_overload = 0

[Tidy]
tidy.clean_output = Off

[soap]
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400
soap.wsdl_cache_limit = 5
[ldap]
ldap.max_links = -1

Answer 1

Heres whats happening.

It says in your error log server reached pm.max_children setting (15), consider raising it

So the max.children limit of 15 means PHP-FPM will stop launching processes once an app has 15 processes running, and any more requests for processes which come in will be queued until one of the previous processes ends.

You are using a php script to generate a 404 page, you then load a page with a load of broken links, your Nginx try files directive ends with a php script:

try_files $uri $uri/ /index.php?$args;

From the Nginx docs that means:

If none of the files were found, an internal redirect to the uri specified in the last parameter is made.

So for every broken link you just added an extra php process to the queue. If you count your 502 errors in the log you'll see there are 15. Because Nginx looks for 15 /index.php?$args which it can't find so tries to display a 404 which guess what? Is generated in php and now everything is broken.

15 processes which cant return 404 because the process limit has been reached and they each need another process to generate a 404 page, so until they time out no more processes for you.

The whole idea of serving a 404 page this way is crazy anyway. It's a static page, you should be serving it from Nginx because web servers are really really good at delivering static content fast, passing it to php, which in turn requests it from your own server again makes absolutely no sense.

Download your custom page to a file:

curl -o /var/www/vuyk.eu/webroot/404.html https://test.vuyk.eu/404-page-not-found

Now add an error page directive in your Nginx conf:

error_page 404 /404.html;

and now you have Nginx serving a custom error page without changing the client url and absolutely no load on your server.