Performance tuning CakePHP application

Question

I just got this quite large CakePHP app (about 20k lines of code), which isn't very clean and there is no documentation at all. The app is running in production, but it has really major problems with performance.

Server is Quad core with 8GB RAM, but the app can serve only about 3-4 requests/s, which is very very bad. Each request takes about 20-30% of all four CPUs.

When I try even little load test like ab -n 100 -c 10 ..., it goes up to 7000ms average response. However, I never made it over 800MB RAM, so there is at least 6GB free RAM for some tweaking.

The problem is, that I haven't been yet able to create working development instance, so I have to tune it in production ...

What do you recommend for easy boosting the performance, without too much digging into source code?

Answer 1

Step 1: Make sure it's the application, and not the web server

Create a simple hello world file outside the Cake hierarchy

<?php
echo 'Hello World';

And see how long that takes to run. Sometimes it's easy to blame the application for something that's going on on the server/network level.

Assuming test.php renders in a reasonable amount of time, move on to step two.

Step 2: Back Everything Up

Fiddling with production code is always a dangerous game. Before you start do a full database backup in case you corrupt something beyond repair, and copy the entire cake directory tree. Whenever you're done for the day, diff the contents of the production directory and your copy (using a GUI tool or the command line)

diff -r production-cake copy-of-cake

Step 3: The Database is Almost Always your First Bottleneck With the LAMP Stack

PHP applications generate a lot of SQL queries, particularly when people are using an ActiveRecord style model which hides a lot of the actual SQL querying. You'll want to set Cake up to log queries to a file and/or to a database table. There's some instructions here on doing this, although I'd recommend logging out to a flat file and/or the syslog instead of the database. Logging DB requests to the database will double the number of queries per page load.

I'd also recommend adding in an IP check so it only logs requests coming from your IP address. That way your logging doesn't dramatically interfere with the regular running of the application.

Once this is in place, make a single request and then look at the SQL that's being generated. Look for identical queries being repeated over and over again as a place where you can drop in some caching to get a performance boost. Also look for sequential queries

select * from foo where id = 5
select * from foo where id = 6
etc...

Which indicate someone's loading up models in a loop without understanding what's going on behind the scenes.

Step 4: If its not the Database, it's System Calls

If the database isn't yoru bottleneck and PHP/Apache are functioning properly, the next thing to look for is system calls. Shelling out is a quick and dirty way to get things done, but its a hugely expensive operation. Get one or two of those in a loop and you're done for.

Run top or ps on your production server and look for programs that are starting and stopping, then search through the code base for those commands.

Step 4: Copy Every Controller

You're going to have a number of controllers

/app/controllers/posts_controller.php
/app/controllers/other_controller.php
etc...

which will correspond to URLs

http://www.example.com/posts/methodName
http://www.example.com/other/methodName
etc...

Whenever you need to debug a particular request to figure out why it's so slow, make a copy of the controller.

/app/controllers/debugposts_controller.php

and manually make the request

http://www.example.com/debugposts/methodName

Then you can throw as many debug/print statements as you want into the controller file. If you're "lucky", the original developers probably jammed a lot of logic in the controller files. If that's the situation, you're now in a position to play the "comment out half the code" game.

Answer 2

You can set the DEBUG level in app/config/core.php and see what is going on. The downside is that so can all your users. With debugging on you'll easily be able to see the slow queries. Barring that, turn on the SQL Slow Query Log and set the cutoff relatively low (because Cake likes to do many, many queries to answer seemingly simple questions).