Readline is too slow - Anything Faster?

Question

I am reading in from a stream using a BufferedReader and InputStreamReader to create one long string that gets created from the readers. It gets up to over 100,000 lines and then throws a 500 error (call failed on the server). I am not sure what is the problem, is there anything faster than this method? It works when the lines are in the thousands but i am working with large data sets.

BufferedReader in = new BufferedReader(new InputStreamReader(newConnect.getInputStream()));
String inputLine;               
String xmlObject = "";
StringBuffer str = new StringBuffer();

while ((inputLine = in.readLine()) != null) {
    str.append(inputLine);
    str.toString();
}       
in.close();

Thanks in advance

Answer 1

to create one long string that gets created from the readers.

Are you by any chance doing this to create your "long string"?

String string;
while(...) 
 string+=whateverComesFromTheSocket;

If yes, then change it to

StringBuilder str = new StringBuilder(); //Edit:Just changed StringBuffer to StringBuilder
while(...)
 str.append(whateverComesFromTheSocket);
String string = str.toString(); 

String objects are immutable and when you do str+="something", memory is reallocated and str+"something" is copied to that newly allocated area. This is a costly operation and running it 51,000 times is an extremely bad thing to do.

StringBuffer and StringBuilder are String's mutable brothers and StringBuilder, being non-concurrent is more efficient than StringBuffer.

Answer 2

readline() can read at about 90 MB/s, its what you are doing with the data read which is slow. BTW readline removes newlines so this approach you are using is flawed as it will turn everying into one line.

Rather than re-inventing the wheel I would suggest you try FileUtils.readLineToString() This will read a file as a STring without discarding newlines, efficiently.