HTML article content extraction - Alchemy API alternative

Question

I've been doing a lot of research to figure out the best way to code an application to get the main article content from almost any HTML webpage. I have a C program that uses libxml2 to parse through the XML, but I came across Alchemy API, which appears to do what I want.

However, it only has an online API and I wanted to keep the application in-house without relying on any external calls.

So does anybody have tips? I was hoping for an off-line alternative that does what Alchemy API can do (paid/non-paid).

My alternative may be to just parse the HTML and use NLP (Natural Language Processing) techniques and other methods to get at the main article content. The types of websites that it will be used include websites with a news section or a blog.

Answer 1

there are a few open source tools available that do similar article extraction tasks. https://github.com/jiminoc/goose which was open source by Gravity.com

It has info on the wiki as well as the source you can view. There are dozens of unit tests that show the text extracted from various articles.