Failed requests by length in my ApacheBench load test result

Question

I have a website in PHP, Lighttpd. It uses also MySQL on Centos 5. I've tested my PHP with code below with Apache Bench (ab). It resulted in some errors (Failed Requests) indicating other length than normal. I'm absolutely sure that my PHP result should always have the same exact length. I've reviewed my Lighttpd and MySQL logs and error logs and don't have any errors there.

Is there any way to check exactly what ab gets when result has other length or is there any other way to find out what is the cause or what is the "bad" result?

I need to know that because I need to have 100% good results.

-bash-3.2# ab -n 500 -c 200 http://domain.com/test/index.php
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking domain.com (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Finished 500 requests


Server Software:        lighttpd/1.4.20
Server Hostname:        domain.com
Server Port:            80

Document Path:          /test/index.php
Document Length:        15673 bytes

Concurrency Level:      200
Time taken for tests:   0.375862 seconds
Complete requests:      500
Failed requests:        499
   (Connect: 0, Length: 499, Exceptions: 0)
Write errors:           0
Total transferred:      7920671 bytes
HTML transferred:       7837000 bytes
Requests per second:    1330.28 [#/sec] (mean)
Time per request:       150.345 [ms] (mean)
Time per request:       0.752 [ms] (mean, across all concurrent requests)
Transfer rate:          20579.36 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   10   9.4      6      30
Processing:     0  113 133.5     16     342
Waiting:        0  111 134.3     12     341
Total:          0  123 138.9     16     370

Percentage of the requests served within a certain time (ms)
  50%     16
  66%    235
  75%    289
  80%    298
  90%    331
  95%    345
  98%    365
  99%    368
 100%    370 (longest request)

Answer 1

Run ab with the -v 2 parameter, meaning verbosity level 2. This will dump the response headers. If your requests are not using chunked encoding, you will see a "Content-Length" header indicating the size of each response.

gw:~$ ab -n 1 -v 2 "http://whatever.com/"  ...  LOG: header received: HTTP/1.0 200 OK ... Content-Length: 1568399 

If your responses use chunked encoding, then the length is not known until the transfer ends. Usually chunked encoding is only used for compressed responses, and ApacheBench doesn't do compression by default.

If it is compressing the responses for whatever reason that might explain it; the compressed length depends on the content.

You can also use curl -i and the --compress option to see the response headers to a single request with and without compression.

Answer 2

Use tcpdump

Open qty 2 terminal/shell windows or just use screen.

In the first window, use tcpdump to capture transmission data from/to your NIC (eth0) to a file:

sudo tcpdump -s 9999 -i eth0 -w myfile.txt

In the second window, fire off your ab command:

ab -n 500 -c 200 http://domain.com/test/index.php

When that's all done, parse the file with strings and grep:

strings myfile2.txt | grep -C 3 "200 OK"

You should be able to monitor all the data segments from there by eyeballing or grep'ing the results.

Answer 3

ab assumes that all responses are the same. It looks at the content-length of the first response, and then compares others to that.

From the man page:

Document Length
  This is the size in bytes of the first successfully returned document. 
  If the document length changes during  testing,  the  response  is 
  considered an error.

So if your first request contains following data:

{"hostname":"nodecellar-1-dwfxd","serverip":"10.1.3.3"}

And the next one is:

{"hostname":"nodecellar-1-dwfxd","serverip":"10.1.3.30"}

ab will fail with a Length error, since the output is one character longer.