Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

To many long running Apache processes in READ status without requests after a certain time

I am using /server-status to monitor the Apache processes. When starting they look like this:

_____W_K__K____________C_K________C_____________W_..............
................................................................
................................................................

But after few hours of running look like this:

R_KCR___KR__RKRR_RRRKRRRRRRKRR_RRCK____R_RRRR_RRRKRRRKRRRRRRRRR_
R_RRRR_R.RR.R_R.R_R..CKRRRRW.K_RCRKRR_R_.._R._.RK_KRK_.RRR.KK_.R
..RR............................................................

There is too many "Read" (R) status which took a long time and I don't know what they do, because they even have no requests. (Mention that there are additional "." statuses which I skipped from the example above, in total I have 2000 positions available.) In the process list I have approx 40 "R" processes like this one:

Srv  PID   Acc        M CPU  SS   Req Conn Child Slot   Client       VHost Request
15-2 21291 0/37/11158 R 0.03 7468 2   0.0  1.93  198.35 82.78.95.105

The report's header look like this:

Server Version: Apache/2.4.10 (Debian) mod_fcgid/2.3.9 OpenSSL/1.0.1t
Server MPM: prefork
Server Built: Sep 15 2016 20:44:43
Current Time: Thursday, 12-Jan-2017 08:38:46 EET
Restart Time: Wednesday, 11-Jan-2017 00:51:18 EET
Parent Server Config. Generation: 3
Parent Server MPM Generation: 2
Server uptime: 1 day 7 hours 47 minutes 27 seconds
Server load: 0.34 0.35 0.39
Total accesses: 1599556 - Total Traffic: 29.9 GB
CPU Usage: u18.87 s6.81 cu0 cs0 - .0224% CPU load
14 requests/sec - 274.0 kB/second - 19.6 kB/request
90 requests currently being processed, 27 idle workers

I have a private server with: 4X3.6GHZ Xeon processor, 2X512 GB SSD, 32 GB memory, Debian 3.16.36-1, Apache 2.4.10, PHP 5.6.29, mod_fcgid, php-fpm, php5-cgi, prefork.

Some of my apache2.conf settings:

Timeout 14400
KeepAlive On
MaxKeepAliveRequests 1000
KeepAliveTimeout 3
HostnameLookups Off

My mpm_prefork.conf settings:

<IfModule mpm_prefork_module>
  StartServers 50
  MinSpareServers 25
  MaxSpareServers 50
  ServerLimit 2000
  MaxRequestWorkers 2000
  MaxConnectionsPerChild 1000
</IfModule>

Some of my fcgid.conf settings:

FcgidMaxRequestLen 1073741824
FcgidOutputBufferSize 1073741824
FcgidMaxProcesses 200
FcgidMaxProcessesPerClass 100
FcgidMinProcessesPerClass 0
FcgidProcessLifeTime 30
FcgidConnectTimeout 30
FcgidIOTimeout 14400
FcgidBusyTimeout 14400
FcgidIdleTimeout 3
FcgidIdleScanInterval 1

I have some long runing cronjobs in the servers, this is why I need that big "14400" (=4 hours) timeout on Apache and PHP processes.

Questions:

  1. Why there are so many "R" statuses without requests?
  2. What they do, how can I find out?
  3. Why they run so long?

Update 1:

A have modified the timeouts to 7200 (=2 hours), this is the minimum for me to can run my cronjobs. However I did not have problems with my cronjobs.

FcgidIOTimeout 7200
FcgidBusyTimeout 7200

Also for Apache:

Timeout 7200

Based on PHP and mod_fcgid: ap_pass_brigade failed in handle_request_ipc function I did this:

FcgidOutputBufferSize 0

Based on mod_fcgid: ap_pass_brigade failed in handle_request function I did this:

FcgidMaxRequestsPerProcess 500

After few hours the server stopped responding:

Service Unavailable: The server is temporarily unable to service your request due to maintanance downtime or capacity problems. Please try again later.

In the logs I found some erros like this two:

[fcgid:warn] (104)Connection reset by peer: mod_fcgid: ap_pass_brigade failed in handle_request_ipc function
[fcgid:warn] (32)Broken pipe: mod_fcgid: ap_pass_brigade failed in handle_request_ipc function

And a bunch of like this (maybe for each requests):

[fcgid:warn] mod_fcgid: too much processes, please increase FCGID_MAX_APPLICATION

Questions 2:

  1. What are these errors mean? How can I correct them?
  2. Why the server stops responding? Probably because of errors...
  3. Are the mod_fcgid, php-fpm, php5-cgi, prefork modules fully compatible and effective? This was the default configurations when I got the new server.
like image 918
Buzogany Laszlo Avatar asked Jan 12 '17 07:01

Buzogany Laszlo


People also ask

Why is Apache spawning so many processes?

Because that is how apache works: it is started as root and then spawns 'children' that are started as a normal user. Security wise is is a very good approach since the user apache2 should not be able to brick your system.

How to monitor Apache performance?

Monitoring Apache with native toolsApache has a module called mod_status that provides a fairly in-depth look at what Apache is doing at any given time, including a summary of performance data like current CPU usage and the total number of requests made since the Apache HTTPD Server was started.

How do I turn off Apache server status?

This functionality comes from a standard module called status and so you can simply disable that module to remove the /server-status endpoint. Show activity on this post. #<Location /server-status> # SetHandler server-status # Order deny,allow # Deny from all # Allow from localhost 127.0.

What is Apache busy workers?

This answer is not useful. Show activity on this post. If you have only very few or none idle workers it means Apache is using all the processes it is allowed to use and new incoming requests have to wait for older requests to finish before they can be handled.


1 Answers

Problem solved.

The solution was to lower the Apache's Timeout value to a number like 15. As I realized for running a long PHP script (even for hours) do not need to this timeout to be hight, it's enought that the PHP's max_execution_time to be big enought.

Update

I have also upgraded to PHP 7.1 with FastCGI under PHP-FPM, and I have changed the Apache's MPM mode to event as esra-s suggested, and it runs like hell. Many thanks!

like image 137
Buzogany Laszlo Avatar answered Oct 18 '22 21:10

Buzogany Laszlo