How to benchmark apache/nginx setup

Question

I am planning to setup nginx as reverse proxy. I will have apache to deliver my dynamic content, and nginx will deliver the static content.

My configuration i have now is just Apache with fastCGI. This gives me no configuration problems and runs great.

After I have set up nginx I want to run some benchmarks to see if I really got some performance increases, else i will switch back.

Does anyone know how I can benchmark this type of setup? Or maybe someone did this already and have some canned results, I will be glad to hear them.

PS. I know this is more a serverfault type of question, but i have seen numerous posts about apache and nginx so i thought i give it a try

Answer 1

Better solution? Siege.

More accurate benchmarking tool than ab

Answer 2

A few things:

1. Don't use ab. It's single-threaded and you will probably end up benchmarking ab rather than your HTTP server.

2. Don't run whatever stress tool you use on the same system as the server. The HTTP server will be competing with the tool for CPU and other resources. Plus localhost's idealized version of a network doesn't tell the whole picture (see point 4).

3. Pay attention to memory and CPU utilization during the tests. So many people never consider this factor. Even if both setups perform equally, if one uses a fraction of the RAM/CPU, then you have a winner.

4. RPS isn't the only meaningful metric. Things like slow clients (3G smartphones, congested networks, slow PC's) can have a distinctly negative impact on threaded servers. The idealized lab setup (localhost or isolated switch) won't reflect this.

5. Your FCGI script will be the bottleneck for either server. I'd suggest using a tool that can pull multiple resources (ideally an entire page, including static content) so you can get a complete picture of page load times.

You might consider using one of the "cloud-based" testing tools, such as browsermob.com or loadimpact.com.