HttpClient not able to access a page behind a login page

Question

[Edit - Added analysis from fiddler, added more code to copy over authentication header] [Edit - now use FormUrlEncodedContent]

I have a page here: https://www.cdc.co.nz/products/list.html?cat=5201 that is password protected via login over here: https://www.cdc.co.nz/login/

The code below allows me to login successfully. However, despite using the same client, I am unable to make a call to the page mentioned above (401 Unauthorized)

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;

    var baseAddress = new Uri("https://www.cdc.co.nz");

    var cookieContainer = new CookieContainer();

    using (var handler = new HttpClientHandler() { CookieContainer = cookieContainer, UseCookies = true })

    using (HttpClient client = new HttpClient(handler) { BaseAddress = baseAddress })
    {
        HttpResponseMessage response = null;

        //Let's visit the homepage to set initial cookie values
        Task.Run(async () => response = await client.GetAsync("/")).GetAwaiter().GetResult(); //200

        string urlToPost = "/login/";

        var postData = new List<KeyValuePair<string, string>>();
        postData.Add(new KeyValuePair<string, string>("username", "username"));
        postData.Add(new KeyValuePair<string, string>("password", "password"));

        HttpContent stringContent = new FormUrlEncodedContent(postData);

        client.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
        client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, br");
        client.DefaultRequestHeaders.Add("Accept-Language", "en-GB,en-US;q=0.9,en;q=0.8");
        client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36");

        client.DefaultRequestHeaders.Add("Origin", "https://www.cdc.co.nz");
        client.DefaultRequestHeaders.Add("Upgrade-Insecure-Requests", "1");

        client.DefaultRequestHeaders.Add("Connection", "keep-alive");
        client.DefaultRequestHeaders.Add("Host", "www.cdc.co.nz");

        client.DefaultRequestHeaders.Add("Referer", "https://www.cdc.co.nz/login/");

        cookieContainer.Add(baseAddress, new Cookie("_ga", "GA1.3.720299450.1533761418"));
        cookieContainer.Add(baseAddress, new Cookie("_gat_oldTracker", "1"));
        cookieContainer.Add(baseAddress, new Cookie("_gat", "1"));
        cookieContainer.Add(baseAddress, new Cookie("_gid", "GA1.3.1011102476.1533761418"));




//Tyler's suggestion here works! 
            //cookieContainer.Add(baseAddress, new Cookie("PHPSESSID", "value from browser login response header"));

        //Receiving 200 response for the nextline, though it returns a 302 in a browser environment
        Task.Run(async () => response = await client.PostAsync(urlToPost, stringContent)).GetAwaiter().GetResult();

        //401 response for the next line
        Task.Run(async () => response = await client.GetAsync("/products/list.html?cat=5201")).GetAwaiter().GetResult();
    }

Fiddler for browser environment: Result: 302 Protocol: HTTPS Host: www.cdc.co.nz URL: /login/

Raw Request Header Browser Environment:

POST /login/ HTTP/1.1
Host: www.cdc.co.nz
Connection: keep-alive
Content-Length: 69
Cache-Control: max-age=0
Origin: https://www.cdc.co.nz
Upgrade-Insecure-Requests: 1
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Referer: https://www.cdc.co.nz/login/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Cookie: _ga=GA1.3.720299450.1533761418; _gid=GA1.3.1011102476.1533761418; PHPSESSID=p3jn5qqhcul59blum597mp2o41; _gat=1; _gat_oldTracker=1

Response Raw Header (Set-Cookie: PHPSESSID=oh7in7n5pjbkrkng4qwwwn22uaq951 is what I am interested) in browser environment:

HTTP/1.1 302 Found
Date: Thu, 09 Aug 2018 00:51:11 GMT
Server: Apache/2.4.7 (Ubuntu)
X-Powered-By: PHP/5.5.9-1ubuntu4.25
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: PHPSESSID=oh7in7n5pjbkrkng4qwwwn22uaq951 <-------- Needed in subsequent Request headers to not 401.
Location: https://www.cdc.co.nz/home/news.html
Content-Length: 0
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html

Fiddler for HttpClient: Result: 200 Protocol: HTTPS Host: www.cdc.co.nz URL: /login/

Raw Header in HttpClient environment:

    GET /login/ HTTP/1.1
    Host: www.cdc.co.nz
    Connection: keep-alive
    Upgrade-Insecure-Requests: 1
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
    Referer: https://www.cdc.co.nz/home/my-account/
    Accept-Encoding: gzip, deflate, br
    Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
    Cookie: _ga=GA1.3.720299450.1533761418; _gid=GA1.3.1011102476.1533761418; _gat=1; _gat_oldTracker=1; PHPSESSID=sdjm7r2jge751jo39mkesqnfl6

Raw Response Header in HttpClient environment (notice how there is no Set-Cookie header / value here?):

HTTP/1.1 200 OK
Date: Thu, 09 Aug 2018 01:11:14 GMT
Server: Apache/2.4.7 (Ubuntu)
X-Powered-By: PHP/5.5.9-1ubuntu4.25
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Keep-Alive: timeout=5, max=98
Connection: Keep-Alive
Content-Type: text/html; charset=UTF-8
Content-Length: 5668

Answer

Adding the extra KV pairs (without even the specification of other unnecessary details) has now got the code working:

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;

    var baseAddress = new Uri("https://www.cdc.co.nz");

    using (HttpClient client = new HttpClient() { BaseAddress = baseAddress })
    {
        HttpResponseMessage response = null;

        //Let's visit the homepage to set initial cookie values
        Task.Run(async () => response = await client.GetAsync("/")).GetAwaiter().GetResult(); //200

        string urlToPost = "/login/";

        var postData = new List<KeyValuePair<string, string>>();
        postData.Add(new KeyValuePair<string, string>("username", "username"));
        postData.Add(new KeyValuePair<string, string>("password", "password"));
        postData.Add(new KeyValuePair<string, string>("returnUrl", "/login/")); <----- To simulate the browser
        postData.Add(new KeyValuePair<string, string>("service", "login")); <----- To simulate the browser

        HttpContent stringContent = new FormUrlEncodedContent(postData);

        //Receiving 200 response for the nextline, though it returns a 302 in a browser environment
        Task.Run(async () => response = await client.PostAsync(urlToPost, stringContent)).GetAwaiter().GetResult();

        //200 response now
        Task.Run(async () => response = await client.GetAsync("/products/list.html?cat=5201")).GetAwaiter().GetResult();
    }

Answer 1

Try to add hidden form values like browser does

    var postData = new List<KeyValuePair<string, string>>();
    postData.Add(new KeyValuePair<string, string>("username", "username"));
    postData.Add(new KeyValuePair<string, string>("password", "password"));
    postData.Add(new KeyValuePair<string, string>("returnUrl", "/login/"));
    postData.Add(new KeyValuePair<string, string>("service", "login"));