How can I ensure that my HttpClient 4.1 does not leak sockets?

Question

My server uses data from an internal web service to construct its response, on a per request basis. I'm using Apache HttpClient 4.1 to make the requests. Each initial request will result in about 30 requests to the web service. Of these, 4 - 8 will end up with sockets stuck in CLOSE_WAIT, which never get released. Eventually these stuck sockets exceed my ulimit and my process runs out of file descriptors.

I don't want to just raise my ulimit (1024), because that will just mask the problem.

The reason I've moved to HttpClient is that java.net.HttpUrlConnection was behaving the same way.

I have tried moving to a SingleClientConnManager per request, and calling client.getConnectionManager().shutdown() on it, but sockets still end up stuck.

Should I be trying to solve this so that I end up with 0 open sockets while there are no running requests, or should I be concentrating on request persistence and pooling?

For clarity I'm including some details which may be relevant:

OS: Ubuntu 10.10

JRE: 1.6.0_22

Language: Scala 2.8

Sample code:

val cleaner = Executors.newScheduledThreadPool(1) 
private val client = {
    val ssl_ctx = SSLContext.getInstance("TLS")
    val managers = Array[TrustManager](TrustingTrustManager)
    ssl_ctx.init(null, managers, new java.security.SecureRandom())
    val sslSf = new org.apache.http.conn.ssl.SSLSocketFactory(ssl_ctx, SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER)
    val schemeRegistry = new SchemeRegistry()
    schemeRegistry.register(new Scheme("https", 443, sslSf))
    val connection = new ThreadSafeClientConnManager(schemeRegistry)
    object clean extends Runnable{ 
        override def run = {
            connection.closeExpiredConnections
            connection.closeIdleConnections(30, SECONDS)
        }
    }
    cleaner.scheduleAtFixedRate(clean,10,10,SECONDS)
    val httpClient = new DefaultHttpClient(connection)
    httpClient.getCredentialsProvider().setCredentials(new AuthScope(AuthScope.ANY), new UsernamePasswordCredentials(username,password))
    httpClient
}
val get = new HttpGet(uri)
val entity = client.execute(get).getEntity
val stream = entity.getContent
val justForTheExample = IOUtils.toString(stream)
stream.close()

Test: netstat -a | grep {myInternalWebServiceName} | grep CLOSE_WAIT

(Lists sockets for my process that are in CLOSE_WAIT state)

Post comment discussion:

This code now demonstrates correct usage.

Answer 1

One needs to pro-actively evict expired / idle connections from the connection pool, as in the blocking I/O model connections cannot react to I/O events unless they are being read from / written to. For details see

http://hc.apache.org/httpcomponents-client-dev/tutorial/html/connmgmt.html#d4e631