HtmlAgilityPack & Selenium Webdriver returns random results

Question

I'm trying to scrape product names from a website. Oddly, I seem to only scrape random 12 items. I've tried both HtmlAgilityPack and with HTTPClient and I get the same random results. Here's my code for HtmlAgilityPack:

using HtmlAgilityPack;
using System.Net.Http;

var url = @"http://www.roots.com/ca/en/men/tops/shirts-and-polos/";
HtmlWeb web = new HtmlWeb();
var doc = web.Load(url, "GET", proxy, new NetworkCredential(PROXY_UID, PROXY_PWD, PROXY_DMN));
var nodes = doc.DocumentNode.Descendants("div")
            .Where(div => div.GetAttributeValue("class", string.Empty) == "product-name")
            .Select(div => div.InnerText.Trim())
            ;

[UPDATE 1] @CodingKuma suggested I try Selenium Webdriver. Here's my code using Selenium Webdriver:

IWebDriver chromeDriver = new ChromeDriver(@"C:\TEMP\Projects\Chrome\chromedriver_win32");
chromeDriver.Url = "http://www.roots.com/ca/en/men/tops/shirts-and-polos/";
var items = chromeDriver.FindElements(By.ClassName("product-name"));
items.Count().Dump();
chromeDriver.Quit();

I tried this code but still no luck. There are over 20 items on that page, but I seem to only get a random 12. How can I scrape all items on that site?

Answer 1

Since the v1.5.0-beta92,

HtmlAgilityPack has a FromBrowser method that allows you to wait until all elements you want are ready.

Documentation: http://html-agility-pack.net/from-browser

string url = "http://html-agility-pack/from-browser";

var web1 = new HtmlWeb();
var doc1 = web1.LoadFromBrowser(url, o =>
{
    var webBrowser = (WebBrowser) o;

    // WAIT until the dynamic text is set
    return !string.IsNullOrEmpty(webBrowser.Document.GetElementById("uiDynamicText").InnerText);
});
var t1 = doc1.DocumentNode.SelectSingleNode("//div[@id='uiDynamicText']").InnerText

var web2 = new HtmlWeb();
var doc2 = web2.LoadFromBrowser(url, html =>
{
    // WAIT until the dynamic text is set
    return !html.Contains("<div id=\"uiDynamicText\"></div>");
});
var t2 = doc2.DocumentNode.SelectSingleNode("//div[@id='uiDynamicText']").InnerText

Console.WriteLine("Text 1: " + t1);
Console.WriteLine("Text 2: " + t2);

The trick here is to find something that tells you when the page is ready since it's impossible for the library to know.