HttpClient memory management

Question

I have an application that has a thread pool (ThreadPoolExecutor) that is handed tasks that each perform a HttpGet operation and read the InputStream into a byte[] to do something with.

After reading the HttpClient docs I came away with the impression that the best way to manage HttpClient connections across multiple threads is to create a single ThreadSafeClientConnManager and share it through out the application.

After implementing this, I am noticing that even after all of the tasks are completed there is still a significant amount of memory still being used by the ThreadSafeClientConnManager.

Looking at the heap dump, this memory is in the form of byte[] arrays. These are not being held by any references that I created. They are being held by pieces of the ThreadSafeClientConnManager and its pool. I am not sure if they are related to the InputStreams or if they are something else.

All of the tasks themselves and their variables are successfully garbage collected.

If I call getConnectionManager().shutdown() on the ThreadSafeClientConnManager then all of the memory is freed just fine. However, I do not want to have to shutdown the connection, because these HttpGet tasks could happen at anytime. I would like to leave it open during the duration of the applications life.

As the HttpGet tasks run, the memory being held grows more and more and can eventually lead to out of memory errors. When the tasks complete, the memory is not released.

How can I ensure the memory is released after the task that was using it is finished?

Here is the code I am using. It is pieced together as best as I code from the HttpClient docs, other questions here on SO and online.

The creation of the HttpClient:

// Create and initialize HTTP parameters
HttpParams params = new BasicHttpParams();
HttpConnectionParams.setConnectionTimeout(params, 40 * 1000);
HttpConnectionParams.setSoTimeout(params, 40 * 1000);
ConnManagerParams.setMaxTotalConnections(params, 100);
HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);

// Create and initialize scheme registry 
SchemeRegistry schemeRegistry = new SchemeRegistry();
schemeRegistry.register( new Scheme("http", PlainSocketFactory.getSocketFactory(), 80));
schemeRegistry.register(new Scheme("https", SSLSocketFactory.getSocketFactory(), 443));

// Create an HttpClient with the ThreadSafeClientConnManager.
// This connection manager must be used if more than one thread will
// be using the HttpClient.
ClientConnectionManager cm = new ThreadSafeClientConnManager(params, schemeRegistry);
mHttpClient = new DefaultHttpClient(cm, params);

Then, the Runnable that performs the HttpGet is pretty much based exactly on the example from the HttpClient examples for Manual connection release. Here is an example of what it looks like:

HttpClient httpclient = getTheSharedThreadSafeClientConnManager(); // Would return the mHttpClient from above
    try {
        HttpGet httpget = new HttpGet("http://www.apache.org/");

        // Execute HTTP request
        System.out.println("executing request " + httpget.getURI());
        HttpResponse response = httpclient.execute(httpget);

        System.out.println("----------------------------------------");
        System.out.println(response.getStatusLine());
        System.out.println("----------------------------------------");

        // Get hold of the response entity
        HttpEntity entity = response.getEntity();

        // If the response does not enclose an entity, there is no need
        // to bother about connection release
        if (entity != null) {
            InputStream instream = entity.getContent();
            try {
                instream.read();
                // do something useful with the response
            } catch (IOException ex) {
                // In case of an IOException the connection will be released
                // back to the connection manager automatically
                throw ex;
            } catch (RuntimeException ex) {
                // In case of an unexpected exception you may want to abort
                // the HTTP request in order to shut down the underlying
                // connection immediately.
                httpget.abort();
                throw ex;
            } finally {
                // Closing the input stream will trigger connection release
                try { instream.close(); } catch (Exception ignore) {}
            }
        }

    }

Is there more you have to do to release the resources per task? I saw in their ThreadSafeClientConnManager example they used a HttpContext, but I can't find any documentation on how to use it. Is that required? If so how do you use it with a ThreadPoolExecutor?

Thanks so much.

Answer 1

Do you ever invoke ClientConnectionManager's releaseConnection(...) or closeExpiredConnections() methods?