Menu
I am running centos 5.5 with 768mb ram. i keep getting server reached MaxClients setting, consider raising the MaxClients setting in the logs also apache runs really slow. when i look at cacti graphs it shows the server is not even using all the resources.. here is the current configuration
<IfModule prefork.c> StartServers 8 MinSpareServers 5 MaxSpareServers 10 ServerLimit 1024 MaxClients 768 MaxRequestsPerChild 4000 </IfModule> <IfModule worker.c> StartServers 2 MaxClients 150 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 0 </IfModule> free -m total used free shared buffers cached Mem: 768 352 415 0 0 37 -/+ buffers/cache: 315 452 Swap: 0 0 0 top - 11:03:54 up 41 days, 11:53, 1 user, load average: 0.05, 0.03, 0.00 Tasks: 35 total, 1 running, 34 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.3%st Mem: 786432k total, 389744k used, 396688k free, 0k buffers Swap: 0k total, 0k used, 0k free, 38284k cached I have tried the following but the server responds very slowly
<IfModule worker.c> #StartServers 2 #MaxClients 150 #MinSpareThreads 25 #MaxSpareThreads 75 #ThreadsPerChild 25 #MaxRequestsPerChild 0 StartServers 20 MaxClients 1024 ServerLimit 1024 MinSpareThreads 128 MaxSpareThreads 768 ThreadsPerChild 64 MaxRequestsPerChild 0 </IfModule> free -m total used free shared buffers cached Mem: 768 324 443 0 0 37 -/+ buffers/cache: 286 481 Swap: 0 0 0 
@regilero
I have updated to
<IfModule prefork.c> StartServers 12 MinSpareServers 12 MaxSpareServers 12 MaxClients 50 MaxRequestsPerChild 300 </IfModule> using top i see
Tasks: 36 total, 1 running, 35 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 786432k total, 613180k used, 173252k free, 0k buffers Swap: 0k total, 0k used, 0k free, 76488k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1 root 20 0 10364 92 60 S 0.0 0.0 1:09.53 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd/808 3 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khelper/808 124 root 16 -4 12620 8 4 S 0.0 0.0 0:00.00 udevd 533 root 20 0 95504 5692 228 S 0.0 0.7 4:02.94 memcached 546 root 20 0 5924 332 276 S 0.0 0.0 6:54.51 syslogd 557 root 20 0 101m 1456 868 S 0.0 0.2 13:18.64 snmpd 570 root 20 0 62640 316 208 S 0.0 0.0 2:39.56 sshd 579 root 20 0 21656 24 20 S 0.0 0.0 0:00.00 xinetd 589 root 20 0 12072 12 8 S 0.0 0.0 0:00.05 mysqld_safe 940 mysql 20 0 559m 164m 3832 S 0.3 21.5 209:33.88 mysqld 1015 root 20 0 20880 200 132 S 0.0 0.0 0:10.48 crond 1023 root 20 0 46748 4 0 S 0.0 0.0 0:00.00 saslauthd 1024 root 20 0 46748 4 0 S 0.0 0.0 0:00.00 saslauthd 3605 root 20 0 62832 2168 636 S 0.0 0.3 0:02.58 sendmail 3613 smmsp 20 0 57712 1648 504 S 0.0 0.2 0:00.01 sendmail 17610 root 20 0 85932 3312 2600 S 0.0 0.4 0:00.02 sshd 17612 mcmap 20 0 86072 1760 1012 S 0.0 0.2 0:00.17 sshd 17613 mcmap 20 0 12076 1656 1292 S 0.0 0.2 0:00.01 bash 17637 root 20 0 45052 1432 1120 S 0.0 0.2 0:00.00 su 17638 root 20 0 12180 1800 1324 S 0.0 0.2 0:00.08 bash 17740 root 20 0 246m 9264 4516 S 0.0 1.2 0:00.19 httpd 18264 apache 20 0 282m 43m 4940 S 0.0 5.7 0:00.56 httpd 18514 apache 20 0 279m 40m 4832 S 0.0 5.3 0:01.47 httpd 18518 apache 20 0 273m 36m 4396 S 0.0 4.7 0:00.45 httpd 18528 apache 20 0 251m 13m 3660 S 0.0 1.8 0:00.41 httpd 18529 apache 20 0 278m 40m 4340 S 0.0 5.3 0:00.99 httpd 18530 apache 20 0 278m 40m 4268 S 0.0 5.3 0:00.67 httpd 18548 apache 20 0 272m 33m 3516 S 0.0 4.4 0:00.28 httpd 18552 apache 20 0 280m 42m 3684 S 0.0 5.5 0:00.48 httpd 18553 apache 20 0 271m 33m 3768 S 0.0 4.3 0:00.45 httpd 18555 apache 20 0 274m 36m 3672 S 0.0 4.7 0:00.58 httpd 18572 apache 20 0 247m 9020 2856 S 0.0 1.1 0:00.01 httpd 18578 apache 20 0 280m 42m 3684 S 0.0 5.6 0:00.76 httpd 18589 apache 20 0 246m 5452 676 S 0.0 0.7 0:00.00 httpd 18588 root 20 0 12624 1216 932 R 0.0 0.2 0:00.06 free -m total used free shared buffers cached Mem: 768 578 189 0 0 74 -/+ buffers/cache: 504 263 Swap: 0 0 0 
Just added current picture of cacti result last 4 hours. busy periods are monday tuesday. So i will wait till next week to see further results of the config change. but it looks like an improvement as before i only had max 10 threads available. Looking at this do you think i can make more improvment?
free -m total used free shared buffers cached Mem: 768 619 148 0 0 49 -/+ buffers/cache: 570 197 Swap: 0 0 0 NEW TEST
On a 2GB Ram VPS box i have now set prefork to
StartServers 20 MinSpareServers 20 MaxSpareServers 20 ServerLimit 256 MaxClients 256 MaxRequestsPerChild 4000 today morning my memcache server died from
Nov 20 09:28:40 vps22899094 kernel: Out of memory: Kill process 12517 (memcached) score 81 or sacrifice child Nov 20 09:28:40 vps22899094 kernel: Killed process 12517, UID 497, (memcached) total-vm:565252kB, anon-rss:42940kB, file-rss:44kB What should the optimal values be to set in apache?
#/etc/sysconfig/memcached
PORT="11211" USER="memcached" MAXCONN="1024" CACHESIZE="1024" OPTIONS="-l 127.0.0.1" /etc/my.cnf
[mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock user=mysql # Disabling symbolic-links is recommended to prevent assorted security risks symbolic-links=0 bind-address=127.0.0.1 #script thread_concurrency=2 query_cache_size = 16M query_cache_type=1 query_cache_limit=5M # MyISAM # #key-buffer-size = 32M #myisam-recover = FORCE,BACKUP # SAFETY # #max-allowed-packet = 16M #max-connect-errors = 1000000 # CACHES AND LIMITS # tmp-table-size = 32M max-heap-table-size = 32M #query-cache-type = 0 #query-cache-size = 0 max-connections = 50 thread-cache-size = 16 #open-files-limit = 65535 #table-definition-cache = 1024 #table-open-cache = 2048 # INNODB # #innodb-flush-method = O_DIRECT #innodb-log-files-in-group = 2 #innodb-log-file-size = 5M #innodb-flush-log-at-trx-commit = 1 #innodb-file-per-table = 1 #innodb-buffer-pool-size = 921M # LOGGING # log-error = /var/log/mysqld.log log-queries-not-using-indexes = 1 slow-query-log = 1 slow-query-log-file = /var/log/mysqld-slow.log [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid 
214
The MaxClients setting determines the number of Apache processes, and thereby the number of client connections, that can be made (pre-requisite: the pre-fork MultiProcessing Modules or MPM). If the MPM worker is running, it will restrict the number of threads that will be available for clients.
Did you consider using nginx (or other event based web server) instead of apache?
nginx shall allow higher number of connections and consume much less resources (as it is event based and does not create separate process per connection). Anyway, you will need some processes, doing real work (like WSGI servers or so) and if they stay on the same server as the front end web server, you only shift the performance problem to a bit different place.
Latest apache version shall allow similar solution (configure it in event based manner), but this is not my area of expertise.
33
When you use Apache with mod_php apache is enforced in prefork mode, and not worker. As, even if php5 is known to support multi-thread, it is also known that some php5 libraries are not behaving very well in multithreaded environments (so you would have a locale call on one thread altering locale on other php threads, for example).
So, if php is not running in cgi way like with php-fpm you have mod_php inside apache and apache in prefork mode. On your tests you have simply commented the prefork settings and increased the worker settings, what you now have is default values for prefork settings and some altered values for the shared ones :
StartServers 20 MinSpareServers 5 MaxSpareServers 10 MaxClients 1024 MaxRequestsPerChild 0 This means you ask apache to start with 20 process, but you tell it that, if there is more than 10 process doing nothing it should reduce this number of children, to stay between 5 and 10 process available. The increase/decrease speed of apache is 1 per minute. So soon you will fall back to the classical situation where you have a fairly low number of free available apache processes (average 2). The average is low because usually you have something like 5 available process, but as soon as the traffic grows they're all used, so there's no process available as apache is very slow in creating new forks. This is certainly increased by the fact your PHP requests seems to be quite long, they do not finish early and the apache forks are not released soon enough to treat another request.
See on the last graphic the small amount of green before the red peak? If you could graph this on a 1 minute basis instead of 5 minutes you would see that this green amount was not big enough to take the incoming traffic without any error message.
Now you set 1024 MaxClients. I guess the cacti graph are not taken after this configuration modification, because with such modification, when no more process are available, apache would continue to fork new children, with a limit of 1024 busy children. Take something like 20MB of RAM per child (or maybe you have a big memory_limit in PHP and allows something like 64MB or 256MB and theses PHP requests are really using more RAM), maybe a DB server... your server is now slowing down because you have only 768MB of RAM. Maybe when apache is trying to initiate the first 20 children you already reach the available RAM limit.
So. a classical way of handling that is to check the amount of memory used by an apache fork (make some top commands while it is running), then find how many parallel request you can handle with this amount of RAM (that mean parallel apache children in prefork mode). Let's say it's 12, for example. Put this number in apache mpm settings this way:
<IfModule prefork.c> StartServers 12 MinSpareServers 12 MaxSpareServers 12 MaxClients 12 MaxRequestsPerChild 300 </IfModule> That means you do not move the number of fork while traffic increase or decrease, because you always want to use all the RAM and be ready for traffic peaks. The 300 means you recyclate each fork after 300 requests, it's better than 0, it means you will not have potential memory leaks issues. MaxClients is set to 12 25 or 50 which is more than 12 to handle the (removed this strange sentende, I can't remember why I said that, if more than 12 requests are incoming the next one will be pushed in the Backlog queue, but you should set MaxClient to your targeted number of processes).ListenBacklog queue, which can enqueue some requests, you may take a bigger queue, but you would get some timeouts maybe
And yes, that means you cannot handle more than 12 parallel requests.
If you want to handle more requests:
If your problem is really traffic peaks, solutions could be available with caches, like a proxy-cache server. If the problem is a random slowness in PHP then... it's an application problem, do you do some HTTP query to another site from PHP, for example?
And finally, as stated by @Jan Vlcinsky you could try nginx, where php will only be available as php-fpm. If you cannot buy RAM and must handle a big traffic that's definitively desserve a test.
Update: About internal dummy connections (if it's your problem, but maybe not).
Check this link and this previous answer. This is 'normal', but if you do not have a simple virtualhost theses requests are maybe hitting your main heavy application, generating slow http queries and preventing regular users to acces your apache processes. They are generated on graceful reload or children managment.
If you do not have a simple basic "It works" default Virtualhost prevent theses requests on your application by some rewrites:
RewriteCond %{HTTP_USER_AGENT} ^.*internal\ dummy\ connection.*$ [NC] RewriteRule .* - [F,L] Update:
Having only one Virtualhost does not protect you from internal dummy connections, it is worst, you are sure now that theses connections are made on your unique Virtualhost. So you should really avoid side effects on your application by using the rewrite rules.
Reading your cacti graphics, it seems your apache is not in prefork mode bug in worker mode. Run httpd -l or apache2 -l on debian, and check if you have worker.c or prefork.c. If you are in worker mode you may encounter some PHP problems in your application, but you should check the worker settings, here is an example:
<IfModule worker.c> StartServers 3 MaxClients 500 MinSpareThreads 75 MaxSpareThreads 250 ThreadsPerChild 25 MaxRequestsPerChild 300 </IfModule> You start 3 processes, each containing 25 threads (so 3*25=75 parallel requests available by default), you allow 75 threads doing nothing, as soon as one thread is used a new process is forked, adding 25 more threads. And when you have more than 250 threads doing nothing (10 processes) some process are killed. You must adjust theses settings with your memory. Here you allow 500 parallel process (that's 20 process of 25 threads). Your usage is maybe more:
<IfModule worker.c> StartServers 2 MaxClients 250 MinSpareThreads 50 MaxSpareThreads 150 ThreadsPerChild 25 MaxRequestsPerChild 300 </IfModule> If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
