How do I know which cookie(s) are must to make a correct HttpWebRequest?

Question

I am working on a download manager and trying to get cookie required contents using HttpWebRequest. I want to integrate my application to Chrome and so I can get the necessary cookie headers and values from the browser.

But first I need to know if cookie is required to get a content to download and which cookies are they. I can't find any helpful resource about this topic.

This is what I imagine:

HttpWebRequest req = (WebRequest.Create(url)) as HttpWebRequest;
//At first, get if cookies are necessary?
//If it is, get the required cookie headers 
//Then add the cookies to the request
CookieContainer cc = new CookieContainer();
Cookie c1 = new Cookie("header1", "value1");
Cookie c2 = new Cookie("header2", "value2");
CookieCollection ccollection = new CookieCollection();
ccollection.Add(c1);
ccollection.Add(c2);
cc.Add(uri, ccollection);
req.CookieContainer = cc;
//Get response and other stuff......

How can I do these steps?

Answer 1

The cookies required to get content from a server are specified by that server in the HTTP response's "Set-Cookie" header. The generic scenario is:

1. Client makes an HTTP request to the server (this could be a login page, or a download page)
2. Server responds with HTTP response that contains "Set-Cookie" header(s)
3. Client remembers all those cookies
4. Client uses the cookies stored in step 3 for all subsequent requests to the same server

Now, considering your scenario of integrating into Chrome, I imagine that the initial requests (steps 1 to 3) will not be done by your application, but by the Chrome itself. The cookies will be stored in the Chrome's cookie store. So what your application will need to do is to get from Chrome all cookies for the domain where you want to download from, and include those cookies in your request (step 4).

See chrome.cookies document on how to use Chrome API to interact with its cookie store, and Set-Cookie docs from Mozilla for the in-depth description of how cookies are specified in the HTTP response.

Answer 2

Try to capture the cookies from first request (a login page may be) and add all those in next request (download request). Something like below.

public void MakeRequest()
    {
        var container = new CookieContainer();

        HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://example.com/loginpage");
        request.Method = WebRequestMethods.Http.Get;
        request.CookieContainer = container;

        HttpWebResponse response = null;

        response = (HttpWebResponse)request.GetResponse();
        //once you read response u need to add all cookie sent in header to the 'container' so that it can be forwarded on second response
        foreach (Cookie cookie in response.Cookies)
        {
            container.Add(cookie);
        }

        HttpWebRequest downRequest = (HttpWebRequest)WebRequest.Create("http://example.com/downloadpage");
        downRequest.Method = WebRequestMethods.Http.Get;
        downRequest.Proxy = null;

        //As you have added the cookies, this must response fine now
        downRequest.CookieContainer = container;
        response = (HttpWebResponse)downRequest.GetResponse();
        var stream = response.GetResponseStream();
    }

Hope this helps.