Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Get web page content to String is very slow

I did the download a web page with the HttpURLConnection.getInputStream() and to get the content to a String, i do the following method:

String content="";
isr = new InputStreamReader(pageContent);
br = new BufferedReader(isr);
try {
    do {
            line = br.readLine();
            content += line;
        } while (line != null);
        return content;
    } catch (Exception e) {
        System.out.println("Error: " + e);
        return null;
    }

The download of the page is fast, but the processing to get the content to String is very slow. There is another way faster to get the content to a String?

I transform it to String to insert in the database.

like image 773
Renato Dinhani Avatar asked May 06 '11 17:05

Renato Dinhani


2 Answers

Read into buffer by number of bytes, not something arbitrary like lines. That alone should be a good start to speeding this up, as the reader will not have to find the line end.

like image 70
Gregory A Beamer Avatar answered Nov 04 '22 22:11

Gregory A Beamer


Use a StringBuffer instead.

Edit for an example:

StringBuffer buffer=new StringBuffer();

for(int i=0;i<20;++i)
  buffer.append(i.toString());

String result=buffer.toString();
like image 35
Blindy Avatar answered Nov 04 '22 20:11

Blindy