Python - Ordered Headers HTTP Requests

Question

I am currently using python 2.7 requests library and there is no support for ordered headers. I can put ordered data for post and get (like an ordered dictionary) but there is simply no support for headers. Not even in python 3

I know HTTP protocol RFC, indicates that order of headers is insignificant, but the problem is that the 3rd party service I am implementing with doesn't work unless the headers are in order. I know this because I have implemented ordered headers requests in other languages and it works (like java) and yes I am 100% certain of that, because I inspected on burp, and wireshark to make sure that this is the only difference between the requests. But I already have 5,000+ lines in python so migrating there is such a painful decision because of such a problem.

The only solution I have thought is to implement http protocol on top of TCP myself, but this is not a smart solution. I can't have the same quality of code as available solutions and it is a possible point of failure for my code.

See a simplified code example I have below:

data=(("param1","something"),
("param2","something_else"))

headers={'id': 'some_random_number',
'version':'some_random_number' ,
'signature':'some_random_number' ,
                    'Content-Type':'application/x-www-form-urlencoded' ,
                    'charset':'utf-8' ,
                    'Content-Length':str(len(urllib.urlencode(data))) ,
                    'name':'random' ,
                    'User-Agent':'Firefox' ,
                    'Connection':'Keep-Alive' ,
                    'Accept-Encoding':'gzip'}

requests.post("myservice.com",headers=headers, data=data)

The order of the request headers is send like that (not actual order, just an example to get my point across)

'version':'some_random_number' 
'Accept-Encoding':'gzip'
'id': 'some_random_number'
'User-Agent':'Firefox' 
'signature':'some_random_number' 
'Connection':'Keep-Alive' 
'Content-Type':'application/x-www-form-urlencoded'
'charset':'utf-8'
'name':'random' 

Which is a problem for me. I don't know what to do at this point. Any help greatly appreciated. I tried urllib library no support

Answer 1

Expanding on the comment, here is a very, very simple OrderedHeaders that requests might be happy with:

class OrderedHeaders(object):

    def __init__(self, *headers):
        self.headers = headers

    def items(self):
        return iter(self.headers)


oh = OrderedHeaders(('Accept-Charset', 'Foo'), ('Bar', 'Foobar'))

for k, v in oh.items():
    print("%s:%s" % (k, v))

Here is a more verbose example that uses topological sorting to determine which headers must be given before other headers. It requires a little more code, yet you can clearly state what sorting your headers must have once and use the class just like any other dict afterwards.

import sys
import toposort

class OrderedHeaders(dict):
    # The precedence of headers is determined once. In this example, 
    # 'Accept-Encoding' must be sorted behind 'User-Agent'
    # (if defined) and 'version' must be sorted behind both
    # 'Accept-Encoding' and 'Connection' (if defined).
    PRECEDENCE = toposort.toposort_flatten({'Accept-Encoding': {'User-Agent'},
                                            'version': {'Accept-Encoding',
                                                        'Connection'}})

    def items(self):
        s = []
        for k, v in dict.items(self):
            try:
                prec = self.PRECEDENCE.index(k)
            except ValueError:
                # no defined sort for this header, so we put it behind
                # any other sorted header
                prec = sys.maxsize
            s.append((prec, k, v))
        return ((k, v) for prec, k, v in sorted(s))

# Initialize like a dict
headers = OrderedHeaders(name='random', Connection='Keep-Alive')
...
# Setting more values
headers['Accept-Encoding'] = 'gzip'
headers['version'] = '0.1'
headers['User-Agent'] = 'Firefox'
...
# Headers come out of '.items()' like they should
for k, v in headers.items():
    print("%s: %s" % (k, v))

prints

Connection: Keep-Alive
User-Agent: Firefox
Accept-Encoding: gzip
version: 0.1
name: random

because Connection needs to come before version, User-Agent needs to come before Accept-Encoding, Accept-Encoding needs to come before version and name has no sorting and is therefor put last.

You can set values on OrderedHeaders in any order you want, sorting is done in .items(). However you can be sure that a sound ordering is always possible: If you make a mistake and define a circular dependency (e.g. 'version' > 'User-Agent' > 'version'), you'll get a toposort.CircularDependencyError at "compile-time".