Get all links on html page?

Question

Im working on a little hobby project. I already have written the code to get a url, download the header and return the mime type / content type.

However, the step before this is the one im stuck on - i need to retrieve the contents of all the urls on the page based inside a tag, and in quotes i.e.

...
<link rel='shortcut icon' href="/static/favicon.ico" type="image/x-icon" />
...

Would find the favicon link.

Is there anything helpful in the .net library or is this going to have to be a case for regex?

Answer 1

I'd look at using the Html Agility Pack.

Here's an example straight from their examples page on how to find all the links in a page:

 HtmlWeb hw = new HtmlWeb();  HtmlDocument doc = hw.Load(/* url */);  foreach(HtmlNode link in doc.DocumentNode.SelectNodes("//a[@href]"))  {   } 

Answer 2

You need to use the HTML Agility Pack.

For example:

var doc = new HtmlWeb().Load(url);
var linkTags = doc.DocumentNode.Descendants("link");
var linkedPages = doc.DocumentNode.Descendants("a")
                                  .Select(a => a.GetAttributeValue("href", null))
                                  .Where(u => !String.IsNullOrEmpty(u));

Answer 3

There isn't anything built into the BCL, but fortunately you can use the HTML Agility Pack to accomplish this task quite simply.

As for your specific problem, please see Easily extracting links from a snippet of html with HtmlAgilityPack:

private List<string> ExtractAllAHrefTags(HtmlDocument htmlSnippet)
{
    List<string> hrefTags = new List<string>();

    foreach (HtmlNode link in htmlSnippet.DocumentNode.SelectNodes("//a[@href]"))
    {
        HtmlAttribute att = link.Attributes["href"];
        hrefTags.Add(att.Value);
    }

    return hrefTags;
}

Answer 4

How about Regex?

<(a|link).*?href=(\"|')(.+?)(\"|').*?>

with flags IgnoreCase and SingleLine

See demo on systemtextregularexpressions.com regex.matches