Problems with Apache servers and A LOT of httpd processes

Question

We have a problem with our Apache server. It's really slow and cpu usage is really high (sometimes showing a constant 100%).

It's on a ubuntu system.

Any ideas what might be causing this?

"top" shows this (there's probably even more httpd processes that didn't fit in the terminal window):

top - 11:33:19 up  1:07,  2 users,  load average: 37.79, 35.90, 29.14 Tasks:  86 total,  10 running,  76 sleeping,   0 stopped,   0 zombieCpu(s): 90.6%us,  9.4%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st Mem:    786432k total,   379584k used,   406848k free,        0k buffersSwap:        0k total,        0k used,        0k free,        0k cached   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                        5507 nobody    16   0 54540  19m 5204 S    4  2.6   0:03.70 httpd                          5653 nobody    17   0 59996  24m 4808 R    4  3.2   0:01.98 httpd                           3277 nobody    15   0  201m  51m 6724 R    2  6.7   2:14.30 mysqld                         5260 nobody    15   0 53740  18m 4972 S    2  2.4   0:06.53 httpd                           5333 nobody    15   0 54536  19m 5152 S    2  2.5   0:06.50 httpd                          3923 nobody    15   0 52644  18m 6372 S    1  2.4   0:16.80 httpd                           5519 nobody    15   0 51252  15m 4792 S    1  2.0   0:03.02 httpd                          4048 nobody    15   0 52808  17m 5040 R    1  2.3   0:13.86 httpd                           5240 nobody    15   0 53980  18m 4916 S    1  2.4   0:08.42 httpd                           5249 nobody    15   0 52908  17m 5188 S    1  2.3   0:08.23 httpd                          5385 nobody    15   0 52568  16m 4824 S    1  2.2   0:04.19 httpd                           5395 nobody    16   0 52536  16m 4804 S    1  2.1   0:03.90 httpd                          5422 nobody    16   0 51408  15m 4888 S    1  2.1   0:03.49 httpd                           5503 nobody    15   0 52536  16m 4800 S    1  2.1   0:02.67 httpd                          3646 nobody    15   0 52572  16m 5012 S    1  2.2   0:25.42 httpd                           3948 nobody    15   0 52844  17m 5008 R    1  2.3   0:17.15 httpd                          5283 nobody    15   0 54000  18m 4944 S    1  2.4   0:07.05 httpd                           5300 nobody    15   0 54748  19m 5276 S    1  2.6   0:06.20 httpd                          5304 nobody    15   0 52876  17m 4904 S    1  2.2   0:06.45 httpd                           5305 nobody    15   0 52812  17m 4884 S    1  2.2   0:06.02 httpd                          5400 nobody    16   0 52060  16m 4800 R    1  2.1   0:04.10 httpd                           5423 nobody    15   0 54764  19m 5196 S    1  2.6   0:04.37 httpd                          5433 nobody    15   0 52168  16m 4844 R    1  2.1   0:03.38 httpd                           5497 nobody    15   0 52572  16m 4820 S    1  2.2   0:02.83 httpd                          5498 nobody    15   0 53076  17m 4864 S    1  2.3   0:03.14 httpd                           5502 nobody    15   0 51260  15m 4804 S    1  2.0   0:02.70 httpd                          5505 nobody    15   0 54920  19m 4976 S    1  2.5   0:02.68 httpd                           5546 nobody    15   0 52584  16m 4664 S    1  2.2   0:02.48 httpd                          5550 nobody    15   0 51244  15m 4796 S    1  2.0   0:02.42 httpd                           5652 nobody    15   0 51976  17m 5912 S    1  2.3   0:01.24 httpd                          5659 nobody    15   0 55012  19m 4980 S    1  2.6   0:01.25 httpd                           5707 nobody    15   0 51416  15m 4880 S    1  2.1   0:00.09 httpd                          5710 nobody    15   0 51240  15m 4792 S    1  2.0   0:00.14 httpd                           5399 nobody    15   0 52568  16m 4856 S    1  2.2   0:03.86 httpd                          5435 nobody    15   0 52596  16m 4864 S    1  2.2   0:02.96 httpd                           5467 nobody    15   0 52316  16m 4856 R    1  2.1   0:02.93 httpd                          5501 nobody    15   0 52956  17m 4844 S    1  2.2   0:02.79 httpd                           5525 nobody    15   0 54508  19m 4916 S    1  2.5   0:03.41 httpd                          5547 nobody    15   0 52604  18m 5708 S    1  2.3   0:02.77 httpd                           5590 root      15   0 19208 1360 1044 R    1  0.2   0:00.39 top                            5627 nobody    15   0 55648  20m 5164 S    1  2.6   0:02.56 httpd                           5629 nobody    15   0 51248  15m 4792 S    1  2.0   0:01.62 httpd                          5647 nobody    15   0 52956  17m 4884 S    1  2.2   0:01.16 httpd                           5656 nobody    15   0 52588  17m 4884 S    1  2.2   0:00.94 httpd                          5665 nobody    15   0 51244  15m 4792 S    1  2.0   0:01.36 httpd                           5709 nobody    15   0 51928  15m 4628 S    1  2.1   0:00.02 httpd                          5711 nobody    16   0 51148  15m 4600 S    1  2.0   0:00.02 httpd                       

Answer 1

Apache needs to fork a process to handle each connected client. Read more about that here.

The upshot is that you have lots of processes because you have lots of connections. If your server is running slowly as a result then here are a few options:

1. Optimise your application: profile your application's code and try and optimise the areas that place the most load on the server.
2. Consider changing to a lower impact web server, such as nginx. This could increase the number of connections that your server can handle.
3. Add page caching, using something like varnish
4. Add another server, and load balance between them
5. Increase the capabilities of the existing server (e.g. increase RAM, upgrade processor, install a faster hard disk).

All of these options have a cost, in time or money. Unfortunately that's just the way it is :)

The quickest wins in my opinion would probably be options 2 and 3. But it means downtime while they're being installed, or having a backup server to work on.