Apache sockets not closing?

Question

I have a web application written using CherryPy, which is run locally on 127.0.0.1:4321. We use mod-rewrite and mod-proxy to have Apache act as a reverse proxy; Apache also handles our SSL encryption and may eventually be used to transfer all of our static content.

This all works just fine for small workloads. However, I recently used urllib2 to write a stress-testing script that would simulate a workload of 100 clients. After some time, each client gets a 503 error from Apache, indicating that Apache cannot connect to 127.0.0.1:4321. CherryPy is functioning properly, but my Apache error log reveals lines like the following:

[Thu Oct 02 12:55:44 2008] [error] (OS 10048)Only one usage of each socket address (protocol/network address/port) is normally permitted. : proxy: HTTP: attempt to connect to 127.0.0.1:4321 (*) failed

Googling for this error reveals that Apache has probably run out of socket file descriptors. Since I only have 100 clients running, this implies that the connections are not being closed, either between my urllib2 connection and Apache (I am definitely calling .close() on the return value of urlopen), or between Apache and CherryPy.

I've confirmed that my urllib2 request is sending an HTTP Connection: close header, although Apache is configured with KeepAlive On if that matters.

In case it matters, I'm using Python 2.5, Apache 2.2, CherryPy 3.0.3, and the server is running on Windows Server 2003.

So what's my next step to stop this problem?

Answer 1

SetEnv proxy-nokeepalive 1 would probably tell you right away if the problem is keepalive between Apache and CP. See the mod_proxy docs for more info.

Answer 2

You might run the netstat command and see if you have a bunch of sockets in the TIME_WAIT state. Depending on your MaxUserPort setting you might be severly limited in the number of ports available to use. In addition the TcpTimedWaitDelay is usually set to 240 seconds so any sockets that are used cannot be reused for four minutes.

There's more good information here --> http://smallvoid.com/article/winnt-tcpip-max-limit.html