Download js generated html with C#

Question

There is a reports website which content I want to parse in C#. I tried downloading the html with WebClient but then I don't get the complete source since most of it is generated via js when I visit the website.

I tried using WebBrowser but could't get it to work in a console app, even after using Application.Run() and SetApartmentState(ApartmentState.STA).

Is there another way to access this generated html? I also took a look into mshtml but couldn't figure it out.

Thanks

Answer 1

The Javascript is executed by the browser. If your console app gets the JS, then it is working as expected, and what you really need is for your console app to execute the JS code that was downloaded.

Answer 2

You can use a headless browser - XBrowser may server.

If not, try HtmlUnit as described in this blog post.