Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

extracting just page text using HTMLAgilityPack

Ok so i am really new to XPath queries used in HTMLAgilityPack.

So lets consider this page http://health.yahoo.net/articles/healthcare/what-your-favorite-flavor-says-about-you. What i want is to extract just the page content and nothing else.

So for that i first remove script and style tags.

Document = new HtmlDocument();
        Document.LoadHtml(page);
        TempString = new StringBuilder();
        foreach (HtmlNode style in Document.DocumentNode.Descendants("style").ToArray())
        {
            style.Remove();
        }
        foreach (HtmlNode script in Document.DocumentNode.Descendants("script").ToArray())
        {
            script.Remove();
        }

After that i am trying to use //text() to get all the text nodes.

foreach (HtmlTextNode node in Document.DocumentNode.SelectNodes("//text()"))
        {
            TempString.AppendLine(node.InnerText);
        }

However not only i am not getting just text i am also getting numerous /r /n characters.

Please i require a little guidance in this regard.

like image 533
Win Coder Avatar asked Oct 13 '13 08:10

Win Coder


2 Answers

If you consider that script and style nodes only have text nodes for children, you can use this XPath expression to get text nodes that are not in script or style tags, so that you don't need to remove the nodes beforehand:

//*[not(self::script or self::style)]/text()

You can further exclude text nodes that are only whitespace using XPath's normalize-space():

//*[not(self::script or self::style)]/text()[not(normalize-space(.)="")]

or the shorter

//*[not(self::script or self::style)]/text()[normalize-space()]

But you will still get text nodes that may have leading or trailing whitespace. This can be handled in your application as @aL3891 suggests.

like image 153
paul trmbrth Avatar answered Nov 18 '22 22:11

paul trmbrth


If \r \n characters in the final string is the problem, you could just remove them after the fact:

TempString.ToString().Replace("\r", "").Replace("\n", "");
like image 20
aL3891 Avatar answered Nov 19 '22 00:11

aL3891