Java, sockets, BufferedReader, and readline hang ... :(

Question

I'm not a Java programmer at all. I try to avoid it at all costs actually, but it is required that I use it for a class (in the school sense). The teacher requires that we use Socket(), BufferedReader(), PrintWriter() and various other things including BufferedReader()'s readLine() method.

Basically, this is the problem I'm having. The documentation clearly states that readLine should return a null at the end of the input stream, but that's not what's happening.

Socket link       = new Socket(this.address, 80);
BufferedReader in = new BufferedReader( new InputStreamReader( link.getInputStream() ));
PrintWriter   out = new PrintWriter(    new PrintWriter(       link.getOutputStream(), true ));

out.print("GET blah blah blah"); // http request by hand
out.flush(); // send the get please

while( (s=in.readLine()) != null ) {

    // prints the html correctly, hooray!!
    System.out.println(s);
}

Instead of finishing at the end of the HTML, I get a blank line, a 0 and another blank line and then the next in.readLine() hangs forever. Why? Where's my null?

I tried out.close() to see if maybe Yahoo! was doing a persistent http session or something (which I don't think it would without the header that we're willing to do it).

All the Java sockets examples I'm finding on the net seem to indicate the while loop is the correct form. I just don't know enough Java to debug this.

Answer 1

Your problem is the content encoding “chunked”. This is used when the length of the content requested from the web server is not known at the time the response is started. It basically consists of the number of bytes being sent, followed by CRLF, followed by the bytes. The end of a response is signalled by the exact sequence you are seeing. The web server is now waiting for your next request (this is also called “request pipelining”).

You have several possibilities:

• Use HTTP version 1.0. This will cause the webserver to automatically close the connection when a response has been sent completely.
• Specify the “Connection: close” header when sending your request. This will also close the connection.
• Parse content encoding “chunked” correctly and simply treat this as if the response is now complete—which it is.

Answer 2

So you're reading from a socket (you don't show that in your code, but that's what I gather from the text)?

As long as the other side is not closing the connection, Java doesn't know that it's at the end of the input, so readLine() is waiting for the other side to send more data and doesn't return null.