Strict ordering of HTTP headers in HttpWebrequest

Question

In spite of the RFC stating that the order of uniquely-named headers shouldn't matter, the website I'm sending this request to does implement a check on the order of headers.

This works:

GET https://www.thewebsite.com HTTP/1.1
Host: www.thewebsite.com
Connection: keep-alive
Accept: */*
User-Agent: Mozilla/5.0 etc

This doesn't work:

GET https://www.thewebsite.com HTTP/1.1
Accept: */*
User-Agent: Mozilla/5.0 etc
Host: www.thewebsite.com
Connection: keep-alive

The default HttpWebRequest seems to put the Host and Connection headers at the end, before the blank line, rather than just after the url.

Is there any way (using a fork of HttpWebRequest or some other library in Nuget even) to specify the order of headers in a HttpWebRequest?

If possible, I'd rather not start going down the route of implementing a proxy to sort them or having to code the whole thing up using a TcpClient.

I'd appreciate any hints at all on this.

Update: With Fiddler running, header order in HttpWebrequest can be re-shuffled in CustomRules.cs. Still no closer to a solution without a proxy though.

Answer 1

Some server implement header ordering as a precaution for any attacks or spam, an article explaining Why ordering HTTP headers is important.

But the standard is, the order in which header fields with differing field names are received is not significant.

HttpWebRequest, there is no easy way to order the headers and the Connection and Host is added internally.

If ordering is really important, use the HttpClient instead, it can easily arrange the Headers based on the example of @Jason.

If you will be using HttpClient, you can create a custom HttpClientHandler and you can arrange your header from there. It can be something like this.

HANDLER

public class CustomHttpClientHandler : HttpClientHandler
{
    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        request.Headers.Clear();

        request.Headers.Add("Host", $"{request.RequestUri.Authority}");
        request.Headers.Add("Connection", "keep-alive");
        request.Headers.Add("Accept", "*/*");
        request.Headers.Add("User-Agent", "Mozilla/5.0 etc");

        return await base.SendAsync(request, cancellationToken);
    }
}

IMPLEMENTATION

HttpClient clientRequest = new HttpClient(new CustomHttpClientHandler());
await clientRequest.GetAsync(url);

Answer 2

.Net Core

If you set the headers yourself, you can specify the order. When the common headers are added it will find the existing headers instead of appending them:

using System.Net;

namespace ConsoleApp2
{
    class Program
    {
        static void Main(string[] args)
        {
            var request = WebRequest.Create("http://www.google.com");
            request.Headers.Add("Host", "www.google.com");
            // this will be set within GetResponse.
            request.Headers.Add("Connection", "");
            request.Headers.Add("Accept", "*/*");
            request.Headers.Add("User-Agent", "Mozilla/5.0 etc");
            request.GetResponse();
        }
    }
}

Here is an example with HttpClient:

using System.Net.Http;
using System.Threading.Tasks;

namespace ConsoleApp3
{
    class Program
    {
        static async Task Main(string[] args)
        {
            var client = new HttpClient();
            client.DefaultRequestHeaders.Add("Host", "www.google.com");
            client.DefaultRequestHeaders.Add("Connection", "keep-alive");
            client.DefaultRequestHeaders.Add("Accept", "*/*");
            client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 etc");
            await client.GetAsync("http://www.google.com");
            await client.PostAsync("http://www.google.com", new StringContent(""));
        }
    }
}

Edit The above code did not work on .Net Framework only .Net Core

.Net Framework

On .Net Framework the headers are reserved so they cannot be set like this, see Cannot set some HTTP headers when using System.Net.WebRequest.

One work around is to use reflection to modify the behavior of the framework class, but be warned this could break if the libraries are updated so it's not recommended!.

Essentially, HttpWebRequest calls ToString on WebHeaderCollection to serialize. See https://referencesource.microsoft.com/#System/net/System/Net/HttpWebRequest.cs,5079

So a custom class can be made to override ToString. Unfortunately reflection is needed to set the headers as WebRequest copies the collection on assignment to Headers, instead of taking the new reference.

WARNING, THE FOLLOWING CODE CAN BREAK IF FRAMEWORK CHANGES

If you use this, write some unit tests that verify the behavior still stays consistent after updates to .NET Framework

using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Reflection;

namespace ConsoleApp2
{
    class Program
    {
        static void Main(string[] args)
        {
            // WARNING, CODE CAN BREAK IF FRAMEWORK CHANGES
            // If you use this, write some unit tests that verify the behavior still stays consistent after updates to .NET Framework
            var request = (HttpWebRequest)WebRequest.Create("http://www.google.com");
            var field = typeof(HttpWebRequest).GetField("_HttpRequestHeaders", BindingFlags.Instance | BindingFlags.NonPublic);
            var headers = new CustomWebHeaderCollection(new Dictionary<string, string>
            {
                ["Host"] = "www.google.com",
                ["Connection"] = "keep-alive",
                ["Accept"] = "*/*",
                ["User-Agent"] = "Mozilla/5.0 etc"
            });
            field.SetValue(request, headers);
            request.GetResponse();
        }
    }

    internal class CustomWebHeaderCollection : WebHeaderCollection
    {
        private readonly Dictionary<string, string> _customHeaders;

        public CustomWebHeaderCollection(Dictionary<string, string> customHeaders)
        {
            _customHeaders = customHeaders;
        }

        public override string ToString()
        {
            // Could call base.ToString() split on Newline and sort as needed

            var lines = _customHeaders
                .Select(kvp => $"{kvp.Key}: {kvp.Value}")
                // These two new lines are needed after the HTTP header
                .Concat(new [] { string.Empty, string.Empty });

            var headers = string.Join("\r\n", lines);

            return headers;
        }
    }
}