HttpURLConnection does not read the whole response

Question

I use HttpURLConnection to do HTTP POST but I dont always get back the full response. I wanted to debug the problem, but when I step through each line it worked. I thought it must be a timing issue so I added Thread.sleep and it really made my code work, but this is only a temporary workaround. I wonder why is this happening and how to solve. Here is my code:

public static InputStream doPOST(String input, String inputMimeType, String url, Map<String, String> httpHeaders, String expectedMimeType) throws MalformedURLException, IOException {

    URL u = new URL(url);
    URLConnection c = u.openConnection();
    InputStream in = null;
    String mediaType = null;
    if (c instanceof HttpURLConnection) {

        //c.setConnectTimeout(1000000);
        //c.setReadTimeout(1000000);

        HttpURLConnection h = (HttpURLConnection)c;
        h.setRequestMethod("POST");
        //h.setChunkedStreamingMode(-1);
        setAccept(h, expectedMimeType);
        h.setRequestProperty("Content-Type", inputMimeType);

        for(String key: httpHeaders.keySet()) {
            h.setRequestProperty(key, httpHeaders.get(key));

            if (logger.isDebugEnabled()) {
                logger.debug("Request property key : " + key + " / value : " + httpHeaders.get(key));
            }

        }

        h.setDoOutput(true);
        h.connect();

        OutputStream out = h.getOutputStream();

        out.write(input.getBytes());

        out.close();

        mediaType = h.getContentType();

        logger.debug(" ------------------ sleep ------------------ START");
        try {
            Thread.sleep(2000);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        logger.debug(" ------------------ sleep ------------------ END");

        if (h.getResponseCode() < 400) {
            in = h.getInputStream();
        } else {
            in = h.getErrorStream();
        }
    }
    return in;

}

later I do the following to read the input stream

        ByteArrayOutputStream bos = new ByteArrayOutputStream();
        while (is.available() > 0) {
            bos.write(is.read());
        }
        is.close();

        //is.read(bytes);
        if (logger.isDebugEnabled()) {
            logger.debug(" Response lenght is : " + is.available());
            //logger.debug("RAW response is " + new String(bytes));
            logger.debug("RAW response is " + new String(bos.toByteArray()));
        }

It genearates the following HTTP headers

POST /emailauthentication/ HTTP/1.1
Accept: application/xml
Content-Type: application/xml
Authorization: OAuth oauth_consumer_key="b465472b-d872-42b9-030e-4e74b9b60e39",oauth_nonce="YnDb5eepuLm%2Fbs",oauth_signature="dbN%2FWeWs2G00mk%2BX6uIi3thJxlM%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1276524919", oauth_token="", oauth_version="1.0"
User-Agent: Java/1.6.0_20
Host: test:6580
Connection: keep-alive
Content-Length: 1107

In other posts it was suggested to turn off keep-alive by using the

http.keepAlive=false

system property, I tried that and the headers changed to

POST /emailauthentication/ HTTP/1.1
Accept: application/xml
Content-Type: application/xml
Authorization: OAuth oauth_consumer_key="b465472b-d872-42b9-030e-4e74b9b60e39", oauth_nonce="Eaiezrj6X4Ttt0", oauth_signature="ND9fAdZMqbYPR2j%2FXUCZmI90rSI%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1276526608", oauth_token="", oauth_version="1.0"
User-Agent: Java/1.6.0_20
Host: test:6580
Connection: close
Content-Length: 1107

the Connection header is "close" but I still cannot read the whole response. Any idea what do I do wrong?

Answer 1

I think your problem is in this line:

while (is.available() > 0) {

According to the javadoc, available does not block and wait until all data is available, so you might get the first packet and then it will return false. The proper way to read from an InputStream is like this:

int len;
byte[] buffer = new byte[4096];
while (-1 != (len = in.read(buffer))) {
  bos.write(buffer, 0, len);
}

Read will return -1 when there nothing left in the inputstream or the connection is closed, and it will block and wait for the network while doing so. Reading arrays is also much more performant than using single bytes.