How accurate is testing with Siege and AB?

Question

I am curious to know how much I can rely upon the results of load testing with Siege and AB. I realize that they do not take static assets into account (images, JS, CSS), but assuming all that stuff is served from a CDN, if Siege/AB tells me I can cater to 200 concurrent users, is there any reason I shouldn't trust it? Am I not considering any other factors, such as any limitations the machine running the test may have?

Answer 1

Here's a pretty good article that discusses the pitfalls of trying to do Siege or ab type testing and then abstract how well your webapp or website is going to perform under load.

http://www.sonassi.com/knowledge-base/magento-kb/why-siege-isnt-an-accurate-test-tool-for-magento-performance/

Here are some bullets from that page that bring up the problems with doing this type of testing:

• Siege isn’t representative of what a real-user (or multiple users) would actually be doing on your website. It can only load the raw response code and HTML, not all the other elements within a page (images, CSS, JS or other static content) – so effectively, it only tests PHP performance.

• It also has very limited session/cookie support, no support for pipelining and basic support for HTTP/1.1. The load it is generating is nothing like that of a real-user, so whilst its good for a quick reference after changes; it doesn’t really indicate that anything will change for a user in real life.

• Siege is easily fooled, it can’t differentiate between a static file being served (ie. a pure HTML file) or a dynamic file (ie. a dynamic Magento PHP page). So if you are running any kind of static file proxy, the results are immediately skewed. At this point – you’ll only be testing the caching proxy, not the delivery speed behind it.

• So those using Varnish, Nginx caching, mod_pagecache can easily just buffer the page into a cache and you’ll see sub 20ms render times. If your using Varnish, then using Siege to test performance – you might as well be loading an image rather than category URL, as it’ll give the exact same results.

• Testing remote servers is almost pointless as it is a concurrency test (ie. how many requests can be satisfied repeatedly), the immediate bottleneck is the network connection between the two machines. Latency and TCP/IP overheads are what make testing a remote site completely pointless, the slightest network congestion amongst a peer between the two servers will immediately show reduced performance. So, what really starts to come into play is how fast the TCP 3-way handshake can be completed – the server being tested could be serving a dynamic page or static 0 byte file – and you could see exactly the same rates of performance, as connectivity is the bottleneck.

The other issue they discuss in the article which can have the most dramatic impact on your website's overall performance is the latency between your site and the clients that are accessing it. They have a good explanation of how latency can impact your sites performance in ways that your end users will feel it but Siege and ab type of testing tools will never expose.

Again excerpting the article:

Ping from UK to UK

[~]$ ping www.bytemark.co.uk -c4 PING www.bytemark.co.uk (212.110.161.177) 56(84) bytes of data. 64 bytes from extapp-front.bytemark.co.uk (212.110.161.177): icmp_seq=1 ttl=57 time=2.86 ms 64 bytes from extapp-front.bytemark.co.uk (212.110.161.177): icmp_seq=2 ttl=57 time=2.51 ms 64 bytes from extapp-front.bytemark.co.uk (212.110.161.177): icmp_seq=3 ttl=57 time=2.54 ms 64 bytes from extapp-front.bytemark.co.uk (212.110.161.177): icmp_seq=4 ttl=57 time=2.63 ms --- www.bytemark.co.uk ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3005ms rtt min/avg/max/mdev = 2.515/2.641/2.869/0.142 ms 

Ping from UK to USA

[~]$ ping www.mediatemple.net -c 4 PING www.mediatemple.net (64.207.129.182) 56(84) bytes of data. 64 bytes from mediatemple.net (64.207.129.182): icmp_seq=1 ttl=49 time=158 ms 64 bytes from mediatemple.net (64.207.129.182): icmp_seq=2 ttl=49 time=154 ms 64 bytes from mediatemple.net (64.207.129.182): icmp_seq=3 ttl=49 time=154 ms 64 bytes from mediatemple.net (64.207.129.182): icmp_seq=4 ttl=49 time=154 ms  --- www.mediatemple.net ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3004ms rtt min/avg/max/mdev = 154.155/155.282/158.321/1.802 ms 

You can immediately see the difference in performance. For that single TCP/IP connection to the USA from the UK, it took 156ms, 62 times more than to a server in the UK. Which means that before you even try anything, the maximum throughput you can achieve on Siege in a second is going to be around 6 transactions per second, due to latency alone.

I highly suggest reading the entire article. It does a very good job of explaining the pitfalls of doing performance testing using tools such as Siege and ab!