How do you grab a text from webpage (Java)?

Question

I'm planning to write a simple J2SE application to aggregate information from multiple web sources.

The most difficult part, I think, is extraction of meaningful information from web pages, if it isn't available as RSS or Atom feeds. For example, I might want to extract a list of questions from stackoverflow, but I absolutely don't need that huge tag cloud or navbar.

What technique/library would you advice?

Updates/Remarks

• Speed doesn't matter — as long as it can parse about 5MB of HTML in less than 10 minutes.
• It sould be really simple.

Answer 1

You may use HTMLParser (http://htmlparser.sourceforge.net/)in combination with URL#getInputStream() to parse the content of HTML pages hosted on Internet.

Answer 2

You could look at how httpunit does it. They use couple of decent html parsers, one is nekohtml. As far as getting data you can use whats built into the jdk (httpurlconnection), or use apache's

http://hc.apache.org/httpclient-3.x/