Screen scraping over SSL with .NET

Question

What solutions exist for screen scraping a site over SSL for use with .NET?

My use case is that I need to login to a partner website (https), navigate through a dynamic hierarchy, and download a zipped file of reports.

I certainly could use other screen scrapers if there are no good viable options in .NET, either though the framework or OSS.

Answer 1

The gold standard for screen scraping in .NET is the HTML Agility Pack.

As far as retrieving pages over HTTPS, try this article:

• Making Authenticated HTTP Requests from an ASP.NET Page

(As mentioned by other answers, you may actually be after automation rather than screen scraping, in which case you may be better off with WatiN, a framework orginally designed for automated web testing, but plenty flexible enough for what you want)

Answer 2

Perhaps consider WATIN to simulate navigating or WebClient if you can find the items yourself and simulate the logic.