I have a Server running Apache/2.2.22 (Debian), PHP 5.6.17 as FPM and MySQL 5.6.25.
The project runs using a CMS called Redaxo (I don't think it's that important, but I'll tell ya anyway). In Redaxo there are some functions which take some time (e.g. deleting cache and rebuilding it takes 1-2 minutes). In this time, when other users come on the website FPM crashes with a 500 Internal Server Error
and I have to reload the page multiple times until the Server Error disappears and the process is done.
I noticed that this only will happen if too many users are on the site on the same time and only when heavy operations are done.
10 users at the same time just surfing = No Problem
10 users at the same time just surfing, while cache deletion = 500 Error for everyone.
I checked this by disallowing the website for everyone except me (.htaccess deny/allow with ip). Then I did the heavy operation and had no problem. As soon as multiple people were on the site again, the problem was there again.
What could it be? What information do you need from me?
These values are set (not commented) in the php-fpm.conf
[global] pid = /run/php5-fpm.pid error_log = /var/log/php5-fpm.log emergency_restart_threshold = 0 include=/etc/php5/fpm/pool.d/*.conf
These values are set (not commented) in the project specific fpm.conf
[projectname] user = projectname group = projectname listen = /var/run/php5-fpm-projectname.sock listen.owner = projectname listen.group = projectname listen.mode = 0660 pm = dynamic pm.max_children = 150 pm.start_servers = 10 pm.min_spare_servers = 10 pm.max_spare_servers = 30 chdir = / php_value[upload_max_filesize] = 128M php_value[max_post_size] = 128M php_value[max_execution_time] = 180 php_value[memory_limit] = 256M
The script when it fails does much with MySQL and File creation if it helps? But it's pretty big, so I'm not sure if I should post it here? Or if it is even the problem?
The apache error log says either this
[Tue Feb 09 10:54:01 2016] [error] [client {IP}] (104)Connection reset by peer: FastCGI: comm with server "/fcgi-bin-php5-fpm-projectnmae" aborted: read failed [Tue Feb 09 10:54:01 2016] [error] [client {IP}] FastCGI: incomplete headers (0 bytes) received from server "/fcgi-bin-php5-fpm-projectnmae"
or this
[Tue Feb 09 11:00:46 2016] [error] [client {IP}] FastCGI: incomplete headers (0 bytes) received from server "/fcgi-bin-php5-fpm-projectname" [Tue Feb 09 11:00:48 2016] [error] [client {IP}] (104)Connection reset by peer: FastCGI: comm with server "/fcgi-bin-php5-fpm-projectname" aborted: read failed
The fpm-log
says the following. Always different timings of course
[10-Feb-2016 09:40:59] WARNING: [pool projectname] child 10970 exited on signal 7 (SIGBUS) after 50.186611 seconds from start [10-Feb-2016 09:40:59] NOTICE: [pool projectname] child 11092 started
Sometimes there's a warning like this in it
[09-Feb-2016 11:00:41] WARNING: [pool projectname] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 6 total children [09-Feb-2016 11:00:42] WARNING: [pool projectname] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 7 total children
Here are some more debug information
[18-Feb-2016 17:42:01] WARNING: [pool projectname] child 9088 exited on signal 7 (SIGBUS) after 70.130564 seconds from start [18-Feb-2016 17:42:01] NOTICE: [pool projectname] child 9205 started [18-Feb-2016 17:43:55] WARNING: [pool projectname] child 9099 said into stderr: "NOTICE: PHP message: PHP Notice: Undefined offset: 1181 in /var/www/projectname/htdocs/redaxo/include/classes/class.ooarticle.inc.php on line 44" [18-Feb-2016 17:43:55] WARNING: [pool projectname] child 9099 said into stderr: "NOTICE: PHP message: PHP Warning: Invalid argument supplied for foreach() in /var/www/projectname/htdocs/redaxo/include/classes/class.ooredaxo.inc.php on line 134" [18-Feb-2016 17:43:55] WARNING: [pool projectname] child 9099 exited on signal 7 (SIGBUS) after 183.838886 seconds from start [18-Feb-2016 17:43:55] NOTICE: [pool projectname] child 9330 started [18-Feb-2016 17:44:00] WARNING: [pool projectname] child 9101 exited on signal 7 (SIGBUS) after 188.987954 seconds from start [18-Feb-2016 17:44:00] NOTICE: [pool projectname] child 9336 started
Results. You can notice PHP-FPM made our test website almost 350% faster when it comes to loading times. Plus, it made the site twice as resource efficient as it was with mod_php. PHP-FPM, one of the newest way to use PHP in conjunction with a web server, is an alternative PHP FastCGI implementation.
Max_children refers to the maximum number of concurrent PHP-FPM processes allowed to exist in such a pool. If the volume of incoming requests requires the creation of more PHP-FPM processes than the number allowed by the max_children limit, those additional requests are backlogged in a queue to await service.
Conclusion. PHP-FPM is an efficient method on how to minimize the memory consumption and rise the performance for the websites with heavy traffic. It is significantly faster than traditional CGI-based methods in multi-user PHP environments.
This might just be the effect of some lock issue from your MySQL server.
You have to connect to your MySQL host during the latency.
If you can't connect, then you run out of the number of allowed concurrent connections of your MySQL server or your user
If you can connect, you have to see what the mysql command "show processlist" returns. Now you have 2 choices:
Many "Waiting for query cache lock" : this will require that you change some of your MySQL server configuration. (this can be caused by an oversized query cache)
You have a request which takes all the resources, which you will have to optimize.
Unless you have plenty of RAM available (like over 16GB available), I'd suggest you are running out of resources and this is causing the 500 error.
Your configuration is saying that you can spawn up to 150 PHP-FPM processes and each one can use 256MB of memory — this alone enables the PHP-FPM server to use over 38GB of memory, and if that's not available, it will cause the 500 error.
Calculate what each server can use of memory, and then set it properly. Does this CMS need up to 256MB of memory? Could it run with fewer memory (like 32MB)? If MySQL, Apache and Nginx are in this same server, separate the memory each one will use, then set the proper value for pm.max_children
and php_value[memory_limit]
.
Please note that the lack of resources is system-wide, so if your PHP process uses all the memory available, MySQL might end up crashing for running out of resources (this might be the reason for the record not found).
If you can say how much memory you have available, I can help you out in configuring these numbers.
It'd be also good to know how much memory is available before you issue cache deletion and how much is available while it is running — it might be indeed using too much memory and suffocating the other processes (and if it uses PHP-CLI, it may have no memory limit).
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With