Symfony2 website with +100 users = 100% CPU charge

Question

I have a problem I can't solved with my server. I may miss a config somewhere, but I searched and find nothing fine.

Server hardware : Intel Atom N2800 2 cores, 4 threads (http://ark.intel.com/products/58917/) 4Gb RAM

It's running under Debian7 with NGinx, PHP5 FPM and MySQL.

There is only one website running. Average 50 visitors at the same time, peak up to 140 visitors.

This is what happen when a peak time :

top - 23:01:41 up 80 days,  8:40,  1 user,  load average: 19,78, 19,52, 19,37
Tasks: 166 total,  22 running, 144 sleeping,   0 stopped,   0 zombie
%Cpu(s): 90,6 us,  9,1 sy,  0,0 ni,  0,0 id,  0,0 wa,  0,0 hi,  0,3 si,  0,0 st
KiB Mem:   4020300 total,  3232164 used,   788136 free,   362764 buffers
KiB Swap:   525308 total,   121404 used,   403904 free,  1191508 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
32295 www-data  20   0  247m  37m 6456 R  19,5  1,0   4:02.15 php5-fpm
32469 www-data  20   0  251m  40m 6592 R  19,5  1,0   3:30.46 php5-fpm
32689 www-data  20   0  243m  32m 6816 R  19,2  0,8   2:56.03 php5-fpm
32692 www-data  20   0  251m  41m 6564 R  19,2  1,0   2:57.84 php5-fpm
 1115 www-data  20   0  251m  40m 6428 R  18,9  1,0   0:54.01 php5-fpm
 1249 www-data  20   0  251m  40m 6356 R  18,9  1,0   0:42.35 php5-fpm
 1251 www-data  20   0  250m  39m 6420 R  18,9  1,0   0:37.36 php5-fpm
31314 www-data  20   0  251m  40m 6440 R  18,9  1,0   6:52.49 php5-fpm
32296 www-data  20   0  251m  40m 6560 R  18,9  1,0   4:03.63 php5-fpm
32410 www-data  20   0  251m  40m 6708 R  18,9  1,0   3:53.65 php5-fpm
32468 www-data  20   0  247m  36m 6720 R  18,9  0,9   3:27.14 php5-fpm
32471 www-data  20   0  251m  40m 6472 R  18,9  1,0   3:33.24 php5-fpm
32691 www-data  20   0  251m  40m 6584 R  18,9  1,0   3:03.90 php5-fpm
32294 www-data  20   0  251m  40m 6452 R  18,5  1,0   4:18.55 php5-fpm
32297 www-data  20   0  251m  40m 6456 R  18,5  1,0   4:07.43 php5-fpm
32467 www-data  20   0  246m  35m 6456 R  18,5  0,9   3:41.64 php5-fpm
 1250 www-data  20   0  241m  30m 6312 R  18,2  0,8   0:37.06 php5-fpm
32114 www-data  20   0  247m  37m 6428 R  18,2  0,9   4:38.27 php5-fpm
32470 www-data  20   0  243m  32m 6424 R  18,2  0,8   3:30.56 php5-fpm
32527 www-data  20   0  239m  28m 6428 R  18,2  0,7   3:19.96 php5-fpm
 1629 root      20   0 17228 5444 1308 S   2,9  0,1   0:01.63 rkhunter
23561 mysql     20   0  488m  73m 7944 S   1,3  1,9 227:34.60 mysqld
 4016 root      20   0 23740 1660 1136 R   0,7  0,0   0:00.07 top
20368 www-data  20   0 66340 4732 2116 S   0,3  0,1  10:34.08 nginx

You surely understand that I can't let it like that.

The website is built with Symfony2. The config file of symfony is available here : http://pastebin.com/CJbea0Qf

The phpinfo is there : https://drive.google.com/file/d/0ByNmwlJzaeKpNkNTcF9GZGNQVmM/edit?usp=sharing

This is the config of Nginx for the website (I found an example and modify it a little bit) Rewrite rules are the because of old addresses.

server {
    listen *:80;

    server_name ***.fr www.***.fr;

    root /home/www/clients/client1/web7/web/beta/web/;


    rewrite ^/index.php / permanent;
    rewrite ^/disclaimer.php / permanent;
    rewrite ^/disclaimer-mobile.php / permanent;
    rewrite ^/news.php /news permanent;
    rewrite ^/contact.php /contact permanent;

    rewrite ^/app\.php/?(.*)$ /$1 permanent;

    try_files $uri @rewriteapp;

    location @rewriteapp {
            rewrite ^(.*)$ /app.php/$1 last;
    }

    # Deny all . files
    location ~ /\. {
            deny all;
    }

            # Do some static file caching
            location ~* \.(jpg|jpeg|gif|png|css|js|ico|xml)$ {
                    log_not_found     off;
                    expires           1d;
            }

    location ~ ^/(app|app_dev)\.php(/|$) {
            fastcgi_split_path_info ^(.+\.php)(/.*)$;
            include fastcgi_params;
            fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;
            fastcgi_index app.php;
            send_timeout 1800;
            fastcgi_read_timeout 1800;
            fastcgi_buffer_size 128k;
            fastcgi_buffers 256 16k;
            fastcgi_busy_buffers_size 256k;
            fastcgi_pass unix:/var/run/php5-fpm.sock;
            #fastcgi_pass 127.0.0.1:9000;
    }

     location ~ ^/(mystatus|myping)$ {
            access_log off;
            include fastcgi_params;
            fastcgi_pass unix:/var/run/php5-fpm.sock;
    }


    # Statics
    location /(bundles|media) {
            access_log off;
            expires 30d;

            # Font files
            #if ($filename ~* ^.*?\.(eot)|(ttf)|(woff)$){
            #       add_header Access-Control-Allow-Origin *;
            #}

            try_files $uri @rewriteapp;
    }
}

Finally, PHP-FPM config.

pm = dynamic
pm.max_children = 20
pm.start_servers = 10
pm.min_spare_servers = 8
pm.max_spare_servers = 15
pm.max_requests = 400

I don't know if you need other data to help me solve that.

The website is quite "light". Not a lot of user interactions, just simple pages (it's mainly a gallery), so just pages generation.

Thanks a lot

ONE THING THAT MAY BE (or not) IMPORTANT When I reload php5-fpm, the load charge is quite calm (~2), but it amplifies with time until not going under 8.

Evolution I set up a cache with nginx fastcgi_cache parameters. Each page is cached for 5minutes. That will break up all "view" counters but I don't care as the website is clearly faster. I noticed that now, when I do a service php5-fpm reload, the server is going down to 1 in charge ! And it continues like that even with +100 visitors at the same time. But strangely, one time yesterday, it began to go up again, 20 on charge. I did reload php5-fpm and it all went down to 1 ! And it's working on 1 since yesterday. It seems that there some sort of trigger point that makes php-fpm going crazy !

Answer 1

After one week with new settings, here is what I do to improve the performance of my server.

First, I used Nginx fastcgi cache. This clearly improve performance !! The cached pages are very very fast. I followed something like http://www.digitalocean.com/community/articles/how-to-setup-fastcgi-caching-with-nginx-on-your-vps or http://techminded.net/blog/transparent-cache-in-nginx.html It's easy to understand how it works with these. My cache is setted up with 5 minutes. It's too much for some pages, not enough for others. I must work on that point. But as this was PHP-FPM taking all resources, not executing php code has clearly lower the CPU charge.

Second, I set the APC cache size up to 128M And I think that was the biggest problem of my server config. Symfony use APC a lot and the APC cache was 100% fragmented ! I saw this with the apc stat page (got it from here as it was not present on my server : http://jesin.tk/apc-php-download/ ). Now the APC cache is 0.76% fragmented, 95% full, 100% cache hit ! The settings I changed : apc.shm_size 128M and apc.max_file_size 3M.

BTW, on the APC stats page, I can see 1728.50 cache requests/second. Of course, if your cache is bad, the performance will be bad. It's all clear now ! This should be explain somewhere in Symfony's documentation as it's a very important point.

The server is now really stable, CPU load about 2. I think I solved it !