TCP connection is not reused for HTTP requests with HttpURLConnection

Question

I'm have created an application which sends GET requests to a URL, and then downloads the full content of that page.

The client sends a GET to e.g. stackoverflow.com, and forwards the response to a parser, which has the resposibility to find all the sources from the page that needs to be downloaded with subsequent GET requests.

The method below is used to send those GET requests. It is called many times consecutively, with the URLs returned by the parser. Most of those URLs are located on the same host, and should be able to share the TCP connection.

public static void sendGetRequestToSubObject(String RecUrl)
    {
        URL url = new URL(recUrl.toString());
        URLConnection connection = url.openConnection ();
        InputStreamReader isr = new InputStreamReader(connection.getInputStream());
    }

Each time this method is called, a new TCP connection is created (with a TCP 3-way handshake) and the GET is then sent on that connection. But I want to reuse the TCP connections, to improve performance.

I guess that since I create a new URL object each time the method is called, this is the way it going to work...

Maybe someone can help me do this in a better way?

Thanks!

Answer 1

HttpURLConnection will reuse connections if it can!

For this to work, several preconditions need to be fulfilled, mostly on the server side. Those preconditions are described in the article linked to above.

Answer 2

Found the problem! I was not reading the input stream properly. This caused the input stream objects to hang, and they could not be reused.

I only defined it, like this:

InputStreamReader isr = new InputStreamReader(connection.getInputStream());

but I never read from it :-)

I changed the read method as well. Instead of a buffered reader I stole this:

InputStream in = null; 
String queryResult = "";
try {
     URL url = new URL(archiveQuery);
     HttpURLConnection urlConn = (HttpURLConnection) url.openConnection();
     HttpURLConnection httpConn = (HttpURLConnection) urlConn;
     httpConn.setAllowUserInteraction(false);
     httpConn.connect();
     in = httpConn.getInputStream();
     BufferedInputStream bis = new BufferedInputStream(in);
     ByteArrayBuffer baf = new ByteArrayBuffer(50);
     int read = 0;
     int bufSize = 512;
     byte[] buffer = new byte[bufSize];
     while(true){
         read = bis.read(buffer);
         if(read==-1){
           break;
         }
         baf.append(buffer, 0, read);
     }
     queryResult = new String(baf.toByteArray());
     } catch (MalformedURLException e) {
          // DEBUG
          Log.e("DEBUG: ", e.toString());
     } catch (IOException e) {
          // DEBUG
          Log.e("DEBUG: ", e.toString());
     } 
}

From here: Reading HttpURLConnection InputStream - manual buffer or BufferedInputStream?