How to use webclient in a secure site?

Question

I need to automate a process involving a website that is using a login form. I need to capture some data in the pages following the login page.

I know how to screen-scrape normal pages, but not those behind a secure site.

1. Can this be done with the .NET WebClient class?
   • How would I automatically login?
   • How would I keep logged in for the other pages?

Answer 1

One way would be through automating a browser -- you mentioned WebClient, so I'm guessing you might be referring to WebClient in .NET.

Two main points:

• There's nothing special about https related to WebClient - it just works
• Cookies are typically used to carry authentication -- you'll need to capture and replay them

Here's the steps I'd follow:

1. GET the login form, capture the the cookie in the response.
2. Using Xpath and HtmlAgilityPack, find the "input type=hidden" field names and values.
3. POST to login form's action with user name, password, and hidden field values in the request body. Include the cookie in the request headers. Again, capture the cookie in the response.
4. GET the pages you want, again, with the cookie in the request headers.

On step 2, I mention a somewhat complicated method for automating the login. Usually, you can post with username and password directly to the known login form action without getting the initial form or relaying the hidden fields. Some sites have form validation (different from field validation) on their forms which makes this method not work.

HtmlAgilityPack is a .NET library that allows you to turn ill-formed html into an XmlDocument so you can XPath over it. Quite useful.

Finally, you may run into a situation where the form relies on client script to alter the form values before submitting. You may need to simulate this behavior.

Using a tool to view the http traffic for this type of work is extremely helpful - I recommend ieHttpHeaders, Fiddler, or FireBug (net tab).