What is the fastest way to scrape HTML webpage in Android?

Question

I need to extract information from an unstructured web page in Android. The information I want is embedded in a table that doesn't have an id.

<table>  <tr><td>Description</td><td></td><td>I want this field next to the description cell</td></tr>  </table> 

Should I use

• Pattern Matching?
• Use BufferedReader to extract the information?

Or are there faster way to get that information?

Answer 1

I think in this case it makes no sense to look for a fast way to extract the information as there is virtually no performance difference between the methods already suggested in answers when you compare it to the time it will take to download the HTML.

So assuming that by fastest you mean most convenient, readable and maintainable code, I suggest you use a DocumentBuilder to parse the relevant HTML and extract data using XPathExpressions:

Document doc = DocumentBuilderFactory.newInstance()   .newDocumentBuilder().parse(new InputSource(new StringReader(html)));  XPathExpression xpath = XPathFactory.newInstance()   .newXPath().compile("//td[text()=\"Description\"]/following-sibling::td[2]");  String result = (String) xpath.evaluate(doc, XPathConstants.STRING); 

If you happen to retrieve invalid HTML, I recommend to isolate the relevant portion (e.g. using substring(indexOf("<table")..) and if necessary correct remaining HTML errors with String operations before parsing. If this gets too complex however (i.e. very bad HTML), just go with the hacky pattern matching approach as suggested in other answers.

Remarks

• XPath is available since API Level 8 (Android 2.2). If you develop for lower API levels you can use DOM methods and conditionals to navigate to the node you want to extract

Answer 2

The fastest way will be parsing the specific information yourself. You seem to know the HTML structure precisely beforehand. The BufferedReader, String and StringBuilder methods should suffice. Here's a kickoff example which displays the first paragraph of your own question:

public static void main(String... args) throws Exception {     URL url = new URL("http://stackoverflow.com/questions/2971155");     BufferedReader reader = null;     StringBuilder builder = new StringBuilder();     try {         reader = new BufferedReader(new InputStreamReader(url.openStream(), "UTF-8"));         for (String line; (line = reader.readLine()) != null;) {             builder.append(line.trim());         }     } finally {         if (reader != null) try { reader.close(); } catch (IOException logOrIgnore) {}     }      String start = "<div class=\"post-text\"><p>";     String end = "</p>";     String part = builder.substring(builder.indexOf(start) + start.length());     String question = part.substring(0, part.indexOf(end));     System.out.println(question); } 

Parsing is in practically all cases definitely faster than pattern matching. Pattern matching is easier, but there is a certain risk that it may yield unexpected results, certainly when using complex regex patterns.

You can also consider to use a more flexible 3rd party HTML parser instead of writing one yourself. It will not be as fast as parsing yourself with beforehand known information. It will however be more concise and flexible. With decent HTML parsers the difference in speed is pretty negligible. I strongly recommend Jsoup for this. It supports jQuery-like CSS selectors. Extracting the firsrt paragraph of your question would then be as easy as:

public static void main(String... args) throws Exception {     Document document = Jsoup.connect("http://stackoverflow.com/questions/2971155").get();     String question = document.select("#question .post-text p").first().text();     System.out.println(question); } 

It's unclear what web page you're talking about, so I can't give a more detailed example how you could select the specific information from the specific page using Jsoup. If you still can't figure it at your own using Jsoup and CSS selectors, then feel free to post the URL in a comment and I'll suggest how to do it.