Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Extract HTML from URL

I'm using Boilerpipe to extract text from url, using this code:

URL url = new URL("http://www.example.com/some-location/index.html");
String text = ArticleExtractor.INSTANCE.getText(url);

the String text contains just the text of the html page, but I need to extract to whole html code from it.

Is there anyone who used this library and knows how to extract the HTML code?

You can check the demo page for more info on the library.

like image 804
Wassim AZIRAR Avatar asked Mar 06 '11 21:03

Wassim AZIRAR


3 Answers

For something as simple as this you don't really need an external library:

 URL url = new URL("http://www.google.com");
 InputStream is = (InputStream) url.getContent();
 BufferedReader br = new BufferedReader(new InputStreamReader(is));
 String line = null;
 StringBuffer sb = new StringBuffer();
 while((line = br.readLine()) != null){
   sb.append(line);
 }
 String htmlContent = sb.toString();
like image 162
Goran Jovic Avatar answered Sep 21 '22 12:09

Goran Jovic


Just use the KeepEverythingExtractor instead of the ArticleExtractor.

But this is using the wrong tool for the wrong job. What you want is just to download the HTML content of a URL (right?), not extract content. So why use a content extractor?

like image 45
Konrad Rudolph Avatar answered Sep 20 '22 12:09

Konrad Rudolph


With Java 7 and a trick of Scanner, you can do the following:

public static String toHtmlString(URL url) throws IOException {
    Objects.requireNonNull(url, "The url cannot be null.");
    try (InputStream is = url.openStream(); Scanner sc = new Scanner(is)) {
        sc.useDelimiter("\\A");
        if (sc.hasNext()) {
            return sc.next();
        } else {
            return null; // or empty
        }
    }
}
like image 30
Paul Vargas Avatar answered Sep 18 '22 12:09

Paul Vargas