httperf buffer overflow detected when rate is high

Question

I'm running httperf 0.9.0 (downloaded from Google Code) on Ubuntu 12.04.1 LTS 64-bit with 2CPU and 4GB RAM. I'm trying to benchmark web servers but encountered the following buffer overflow problem.

Terminal command:

httperf --timeout=5 --client=0/1 --server=localhost --port=9090 --uri=/?value=benchmarks --rate=1200 --send-buffer=4096 --recv-buffer=16384 --num-conns=5000 --num-calls=10

After running for a few seconds, it crashes:

*** buffer overflow detected ***: httperf terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f1f5efa1007]
/lib/x86_64-linux-gnu/libc.so.6(+0x107f00)[0x7f1f5ef9ff00]
/lib/x86_64-linux-gnu/libc.so.6(+0x108fbe)[0x7f1f5efa0fbe]
httperf[0x404054]
httperf[0x404e9f]
httperf[0x406953]
httperf[0x406bd1]
httperf[0x40639f]
httperf[0x4054d5]
httperf[0x40285e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f1f5eeb976d]
httperf[0x4038f1]
======= Memory map: ========
...
...
7f1f5fd74000-7f1f5fd79000 rw-p 00000000 00:00 0 
7f1f5fd91000-7f1f5fd95000 rw-p 00000000 00:00 0 
7f1f5fd95000-7f1f5fd96000 r--p 00022000 08:03 4849686                    /lib/x86_64-linux-gnu/ld-2.15.so
7f1f5fd96000-7f1f5fd98000 rw-p 00023000 08:03 4849686                    /lib/x86_64-linux-gnu/ld-2.15.so
7fff10452000-7fff10473000 rw-p 00000000 00:00 0                          [stack]
7fff1054f000-7fff10550000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted

I checked the core dump file with gdb as follows:

(gdb) list
198   event_signal (EV_PERF_SAMPLE, 0, callarg);
199 
200   /* prepare for next sample interval: */
201   perf_sample_start = timer_now ();
202   timer_schedule (perf_sample, regarg, RATE_INTERVAL);
203 }
204 
205 int
206 main (int argc, char **argv)
207 {
(gdb) bt
#0  0x00007f33d4643445 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f33d4646bab in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f33d4680e2e in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f33d4716007 in __fortify_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007f33d4714f00 in __chk_fail () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x00007f33d4715fbe in __fdelt_warn () from /lib/x86_64-linux-gnu/libc.so.6
#6  0x0000000000404054 in set_active (s=<optimized out>, fdset=0x612bc0) at core.c:367
#7  0x0000000000404e9f in core_connect (s=0x17e7100) at core.c:980
#8  0x0000000000406953 in make_conn (arg=...) at conn_rate.c:64
#9  0x0000000000406bd1 in tick (t=<optimized out>, arg=...) at rate.c:94
#10 0x000000000040639f in timer_tick () at timer.c:104
#11 0x00000000004054d5 in core_loop () at core.c:1255
#12 0x000000000040285e in main (argc=11, argv=<optimized out>) at httperf.c:971

I tracked the source code a bit and found that FD_SET seems to be the cause.

Finally, for lower rates (e.g. --rate=100 or --rate=500) httperf works fine. I'm benchmarking different web servers, and the rate causing a crash differs. My rates vary from 100 up to 1200.

For more details, actually I'm trying to repeat the experiments done by Roberto Ostinelli, and I have already tuned TCP settings and applied the patch mentioned in his blog post.

Any idea about what's causing this problem? Thanks!

Answer 1

You are trying to use an fd that is > 1024. With low loads you don't need/use that many fds. With high loads you need more fds and eventually get to 1024 and that causes the problem.

Even when I increase __FD_SETSIZE I get this problem, so I think there is actually a bug in whatever code is doing the bound check (gcc/llvm?)

Answer 2

Newer versions of glibc do their own checks internally for FD_SET (called from httperf), and those checks fail, causing an abort. Even though httperf was built with a different __FD_SET_SIZE, glibc was still using the original one it was compiled with.

To get around this, I dug up an old version of sys/select.h and bits/select.h, from before __FD_ELT performed checks, and dropped them into httperf's src/ directory (in sys/ and bits). This way, httperf uses the old FD_SET macros that don't perform the checks that cause the abort. I used glibc-2.14.1, but any version that doesn't have bits/select2.h should do the trick.

A friend of mine is collecting this and other fixes to httperf for our own use (and yours!) at https://github.com/klueska/httperf