C# WebClient login to accounts.google.com

Question

I have very difficult time trying to authenticate to accounts.google.com using webclient

I'm using C# WebClient object to achieve following.

I'm submitting form fields to https://accounts.google.com/ServiceLoginAuth?service=oz

Here is POST Fields:

service=oz
dsh=-8355435623354577691
GALX=33xq1Ma_CKI
timeStmp=
secTok=
[email protected]
Passwd=password
signIn=Sign in
PersistentCookie=yes
rmShown=1

Now when login page loads before I submit data it has following headers:

Content-Type                text/html; charset=UTF-8
Strict-Transport-Security   max-age=2592000; includeSubDomains
Set-Cookie                  GAPS=1:QClFh_dKle5DhcdGwmU3m6FiPqPoqw:SqdLB2u4P2oGjt_x;Path=/;Expires=Sat, 21-Dec-2013 07:31:40 GMT;Secure;HttpOnly
Cache-Control               no-cache, no-store
Pragma                      no-cache
Expires                     Mon, 01-Jan-1990 00:00:00 GMT
X-Frame-Options             Deny
X-Auto-Login                realm=com.google&args=service%3Doz%26continue%3Dhttps%253A%252F%252Faccounts.google.com%252FManageAccount
Content-Encoding            gzip
Transfer-Encoding           chunked
Date                        Thu, 22 Dec 2011 07:31:40 GMT
X-Content-Type-Options      nosniff
X-XSS-Protection            1; mode=block
Server                      GSE

OK now how do I use WebClient Class to include those headers?

I have tried webClient_.Headers.Add(); but it has limited effect and always returns login page.

Below is a class that I use. Would appreciate any help.

---

Getting login page

    public void LoginPageRequest(Account acc)
    {

        var rparams = new RequestParams();
        rparams.URL = @"https://accounts.google.com/ServiceLoginAuth?service=oz";
        rparams.RequestName = "LoginPage";
        rparams.Account = acc;

        webClient_.DownloadDataAsync(new Uri(rparams.URL), rparams);
    }

    void webClient__DownloadDataCompleted(object sender, DownloadDataCompletedEventArgs e)
    {
        RequestParams rparams = (RequestParams)e.UserState;

        if (rparams.RequestName == "LoginPage")
        {
            ParseLoginRequest(e.Result, e.UserState);
        }
    }

Now getting form fields using HtmlAgilityPack and adding them into Parameters collection

    public void ParseLoginRequest(byte[] data, object UserState)
    {
        RequestParams rparams = (RequestParams)UserState;

        rparams.ClearParams();

        ASCIIEncoding encoder = new ASCIIEncoding();

        string html = encoder.GetString(data);

        HtmlNode.ElementsFlags.Remove("form");

        HtmlDocument doc = new HtmlDocument();
        doc.LoadHtml(html);

        HtmlNode form = doc.GetElementbyId("gaia_loginform");

        rparams.URL = form.GetAttributeValue("action", string.Empty);
        rparams.RequestName = "LoginPost";

        var inputs = form.Descendants("input");
        foreach (var element in inputs)
        {
            string name = element.GetAttributeValue("name", "undefined");
            string value = element.GetAttributeValue("value", "");
            if (!name.Equals("undefined")) {

                if (name.ToLower().Equals("email"))
                {
                    value = rparams.Account.Email;
                }
                else if (name.ToLower().Equals("passwd"))
                {
                    value = rparams.Account.Password;
                }

                rparams.AddParam(name,value);
                Console.WriteLine(name + "-" + value);
            }
        }

        webClient_.UploadValuesAsync(new Uri(rparams.URL),"POST", rparams.GetParams,rparams);

After I post the data I get login page rather than redirect or success message.

What am I doing wrong?

Answer 1

After some fiddling around, it looks like the WebClient class is not the best approach to this particular problem.

To achieve following goal I had to jump one level below to WebRequest.

When making WebRequest (HttpWebRequest) and using HttpWebResponse it is possible to set CookieContainer

        webRequest_ = (HttpWebRequest)HttpWebRequest.Create(rparams.URL);

        webRequest_.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)";
        CookieContainer cookieJar = new CookieContainer();
        webRequest_.CookieContainer = cookieJar;

        string html = string.Empty;

        try
        {
            using (WebResponse response = webRequest_.GetResponse())
            {
                using (var streamReader = new StreamReader(response.GetResponseStream()))
                {
                    html = streamReader.ReadToEnd();
                    ParseLoginRequest(html, response,cookieJar);
                }
            }
        }
        catch (WebException e)
        {
            using (WebResponse response = e.Response)
            {
                HttpWebResponse httpResponse = (HttpWebResponse)response;
                Console.WriteLine("Error code: {0}", httpResponse.StatusCode);
                using (var streamReader = new StreamReader(response.GetResponseStream()))
                    Console.WriteLine(html = streamReader.ReadToEnd());
            }
        }

and then when making post use the same Cookie Container in following manner

        webRequest_ = (HttpWebRequest)HttpWebRequest.Create(rparams.URL);

        webRequest_.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)";
        webRequest_.Method = "POST";
        webRequest_.ContentType = "application/x-www-form-urlencoded";
        webRequest_.CookieContainer = cookieJar;

        var parameters = new StringBuilder();

        foreach (var key in rparams.Params)
        {
            parameters.AppendFormat("{0}={1}&",HttpUtility.UrlEncode(key.ToString()),
                HttpUtility.UrlEncode(rparams.Params[key.ToString()]));
        }

        parameters.Length -= 1;

        using (var writer = new StreamWriter(webRequest_.GetRequestStream()))
        {
            writer.Write(parameters.ToString());
        }

        string html = string.Empty;

        using (response = webRequest_.GetResponse())
        {
            using (var streamReader = new StreamReader(response.GetResponseStream()))
            {
                html = streamReader.ReadToEnd();

            }
        }

So this works, this code is not for production use and can be/should be optimized. Treat it just as an example.

Answer 2

This is a quick example written in the answer pane and untested. You will probably need to parse some values out of an initial request for some form values to go in to formData. A lot of my code is based on this type of process unless we need to scrape facebook spokeo type sites in which case the ajax makes us use a different approach.

using System;
using System.Collections.Generic;
using System.Collections.Specialized;
using System.Linq;
using System.Text;

namespace GMailTest
{
    class Program
    {
        private static NameValueCollection formData = new NameValueCollection();
        private static CookieAwareWebClient webClient = new CookieAwareWebClient();

        static void Main(string[] args)
        {
            formData.Clear();
            formData["service"] = "oz";
            formData["dsh"] = "-8355435623354577691";
            formData["GALX"] = "33xq1Ma_CKI";
            formData["timeStmp"] = "";
            formData["secTok"] = "";
            formData["Email"] = "[email protected]";
            formData["Passwd"] = "password";
            formData["signIn"] = "Sign in";
            formData["PersistentCookie"] = "yes";
            formData["rmShown"] = "1";

            byte[] responseBytes = webClient.UploadValues("https://accounts.google.com/ServiceLoginAuth?service=oz", "POST", formData);
            string responseHTML = Encoding.UTF8.GetString(responseBytes);
        }
    }

    public class CookieAwareWebClient : WebClient
    {
        public CookieAwareWebClient() : this(new CookieContainer())
        { }

        public CookieAwareWebClient(CookieContainer c)
        {
            this.CookieContainer = c;
            this.Headers.Add("User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52 Safari/536.5");
        }

        public CookieContainer CookieContainer { get; set; }

        protected override WebRequest GetWebRequest(Uri address)
        {
            WebRequest request = base.GetWebRequest(address);
            if (request is HttpWebRequest)
            {
                (request as HttpWebRequest).CookieContainer = this.CookieContainer;
            }
            return request;
        }
    }
}