Why does a simple Thin server stop responding at 16500 requests when benchmarking? [duplicate]

Question

Possible Duplicate:
'ab' program freezes after lots of requests, why?

Here's a simple test server:

require 'rubygems'
require 'rack'
require 'thin'

class HelloWorld

  def call(env)
    [200, {"Content-Type" => "text/plain"}, "OK"]
  end
end

Rack::Handler::Thin.run HelloWorld.new, :Port => 9294 
#I've tried with these added too, 'rack.multithread' => true, 'rack.multiprocess' => true

Here's a test run:

$ ab -n 20000 http://0.0.0.0:9294/sdf
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 0.0.0.0 (be patient)
Completed 2000 requests
Completed 4000 requests
Completed 6000 requests
Completed 8000 requests
Completed 10000 requests
Completed 12000 requests
Completed 14000 requests
Completed 16000 requests
apr_poll: The timeout specified has expired (70007)
Total of 16347 requests completed

It breaks down at around 16500. Why? How can I find out what's going on. Is it GC in ruby or is it something with number of available network sockets on an OS X machine. I have a MPB 2.5 Ghz 6G memory.

---

Edit

After some discussion here and testing various things, it seems like changing net.inet.tcp.msl from 15000 to 1000ms makes the problem of testing high frequency web servers with ab go away.

sudo sysctl -w net.inet.tcp.msl=1000 # this is only good for local development

See referenced question with the answer to this problem. 'ab' program freezes after lots of requests, why?

Answer 1

I'll add the solution here for claritys sake. The correct solution for managing to do high frequency tests with ab on os X is to change the 'net.inet.tcp.msl' setting from 15000ms to 1000ms. This should only be done on development boxes.

 sudo sysctl -w net.inet.tcp.msl=1000 # this is only good for local development

This answer was found after the good detective work performed in the comments here and comes from an answer to a very similar question here's the answer: https://stackoverflow.com/a/6699135/155031

Answer 2

I think I've got it.

When ab makes connections to your test server, it opens a source port (say, 50134) and makes a connection to the destination port (9294).

The ports that ab opens for the source port are determined by the sysctl settings net.inet.ip.portrange.first and net.inet.ip.portrange.last. For example, on my machine:

philippotter ~ $ sysctl -a | grep ip.portrange
net.inet.ip.portrange.lowfirst: 1023
net.inet.ip.portrange.lowlast: 600
net.inet.ip.portrange.first: 49152
net.inet.ip.portrange.last: 65535
net.inet.ip.portrange.hifirst: 49152
net.inet.ip.portrange.hilast: 65535

This means that ab's source ports will be in the range from 49152 to 65535, which is a total of 16384.

HTTP is a TCP protocol. When a TCP connection is closed, it goes into the TIME_WAIT state, while it waits for any remaining in-transit packets to reach their destinations. This means that the port is not usable for any other purpose until the timeout is reached.

So, putting all of this together, ab uses up all available source ports very quickly; they go into the TIME_WAIT state; they can't be reused; ab is unable to create any more connections.

You can see this if you kill ab when it hangs, and run it again -- it won't be able to create any connections!