investigating apache benchmark failed request

Question

I am just start using AB just today. Read couple of AB tutorial on new and thought give it a try for load testing my site.

After using it couple of time I got huge number of failed request. Can you explain what is mean by a failed request? How can I investigate more for this issue?

Sample AB result:

-jailshell-3.2$ ab -n500 -c10 http://www.tweeting.tv/index.php
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking www.tweeting.tv (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Finished 500 requests


Server Software:        Apache
Server Hostname:        www.tweeting.tv
Server Port:            80

Document Path:          /index.php
Document Length:        242861 bytes

Concurrency Level:      10
Time taken for tests:   97.846330 seconds
Complete requests:      500
Failed requests:        481
   (Connect: 0, Length: 481, Exceptions: 0)
Write errors:           0
Non-2xx responses:      2
Total transferred:      121214449 bytes
HTML transferred:       121003283 bytes
Requests per second:    5.11 [#/sec] (mean)
Time per request:       1956.927 [ms] (mean)
Time per request:       195.693 [ms] (mean, across all concurrent requests)
Transfer rate:          1209.78 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0 1503 1675.5      1    9036
Processing:   130  393 285.1    319    2382
Waiting:       92  244 226.3    192    2180
Total:        153 1896 1726.2   1121   10374

Percentage of the requests served within a certain time (ms)
  50%   1121
  66%   3308
  75%   3355
  80%   3375
  90%   3451
  95%   3603
  98%   4163
  99%   9315
 100%  10374 (longest request)

N.B. I am using Hostgator Linux shared server.

Answer 1

This is an issue with dynamic pages, it happens because the Content-Length can vary between the requests. When using ab with such pages you need to use the -l option.

-l              Accept variable document length (use this for dynamic pages)

Answer 2

The failed requests are all based on length - that is the number of responses that didn't match exactly the byte count stated. This will be due to dynamic content such as adverts etc that are served differently each time so this is nothing to worry about.

Answer 3

Martin is correct. More helpfully, I ran a quick curl against our production servers:

{user@staging:~}$ for i in `seq 1 10`; do curl -sk https://app.copperegg.com/login >    /tmp/lb$i.txt ; done
{user@staging:~}$ wc /tmp/lb*
   74   239  3316 /tmp/lb1.txt
   74   239  3324 /tmp/lb10.txt
   74   239  3320 /tmp/lb2.txt
   74   239  3316 /tmp/lb3.txt
   74   239  3316 /tmp/lb4.txt
   74   239  3316 /tmp/lb5.txt
   74   239  3320 /tmp/lb6.txt
   74   239  3316 /tmp/lb7.txt
   74   239  3316 /tmp/lb8.txt
   74   239  3316 /tmp/lb9.txt
  740  2390 33176 total
{user@staging:~}$ diff /tmp/lb1.txt /tmp/lb10.txt  
7c7
<   var g_time_offset = new Date().getTime() - 1318965621000;
---
>   var g_time_offset = new Date().getTime() - 1318965622000;
15c15
< <meta name="csrf-token" content="bi9pgbUoUQLOwB3V8fT1E40sV06x4914ybLSvnfEeeg="/>
---
> <meta name="csrf-token" content="GL&#47;RRZCf2Zk&#47;AQzRgEW2U4Iv3htD1hodt2qfp4jwIxQ="/>
23c23
<   <form accept-charset="UTF-8" action="/authenticate" id="loginForm" method="post"><div style="margin:0;padding:0;display:inline"><input name="utf8" type="hidden" value="&#x2713;" /><input name="authenticity_token" type="hidden" value="bi9pgbUoUQLOwB3V8fT1E40sV06x4914ybLSvnfEeeg=" /></div>
---
>   <form accept-charset="UTF-8" action="/authenticate" id="loginForm" method="post"><div style="margin:0;padding:0;display:inline"><input name="utf8" type="hidden" value="&#x2713;" /><input name="authenticity_token" type="hidden" value="GL/RRZCf2Zk/AQzRgEW2U4Iv3htD1hodt2qfp4jwIxQ=" /></div>
{user@staging:~}$ diff /tmp/lb1.txt /tmp/lb2.txt 
7c7
<   var g_time_offset = new Date().getTime() - 1318965621000;
---
>   var g_time_offset = new Date().getTime() - 1318965622000;
9,10c9,10
< <link href="/stylesheets/application.css?1318747862" media="screen" rel="stylesheet" type="text/css" />
< <script src="/javascripts/cache/application.js?1318747811" type="text/javascript"></script>
---
> <link href="/stylesheets/application.css?1318747582" media="screen" rel="stylesheet" type="text/css" />
> <script src="/javascripts/cache/application.js?1318747448" type="text/javascript"></script>
15c15
< <meta name="csrf-token" content="bi9pgbUoUQLOwB3V8fT1E40sV06x4914ybLSvnfEeeg="/>
---
> <meta name="csrf-token" content="BMZKKUZ3WFhQrCIewQ81VuArEtUp8gc6ccr0Wi3&#47;sqE="/>
23c23
<   <form accept-charset="UTF-8" action="/authenticate" id="loginForm" method="post"><div style="margin:0;padding:0;display:inline"><input name="utf8" type="hidden" value="&#x2713;" /><input name="authenticity_token" type="hidden" value="bi9pgbUoUQLOwB3V8fT1E40sV06x4914ybLSvnfEeeg=" /></div>
---
>   <form accept-charset="UTF-8" action="/authenticate" id="loginForm" method="post"><div style="margin:0;padding:0;display:inline"><input name="utf8" type="hidden" value="&#x2713;" /><input name="authenticity_token" type="hidden" value="BMZKKUZ3WFhQrCIewQ81VuArEtUp8gc6ccr0Wi3/sqE=" /></div>
{user@staging:~}$ 

Notice that we're seeing a "content" string that has a nice ugly string of characters. Notice that the '/' character is in the "form" line, but in the "meta" line, it's replaced with "/". This accounts for the 4 or 8 character difference between my request lengths.

It's irritating that apachebench isn't smart able to account for this, but at least we can see the cause.