Creating an order-preserving multi-value dict for Django

Question

When attempting to make a cross-compatible order preserving QueryDict subclass:

from collections import OrderedDict

from django.http import QueryDict
from django.conf import settings

settings.configure()

class OrderedQueryDict(QueryDict, OrderedDict):
    pass

querystring = 'z=33&x=11'
print(QueryDict(querystring).urlencode())
print(OrderedQueryDict(querystring).urlencode())

Output on Python 3.x (correct and expected result):

z=33&x=11  # or maybe x=11,z=33 on Python<=3.5
z=33&x=11

Output on Python 2.7 (this querystring was corrupted):

x=11&z=33
z=3&z=3&x=1&x=1

Why does this idea work on Python 3 but not on Python 2?

Django v1.11.20.

Answer 1

TLDR: Re-implement lists:

class OrderedQueryDict(QueryDict, OrderedDict):
    def lists(self):
        """Returns a list of (key, list) pairs."""
        return [(key, self.getlist(key)) for key in self]

For full functionality, iterlists should be re-implemented as well.

---

The problem is that Django's MultiValueDict overwrites __getitem__ to retrieve just the last value, with getlist retrieving all values. This implicitly relies on other methods of the underlying mapping not using overridden methods. For example, it relies on super().iteritems being able to retrieve lists of values:

>>> from django.utils.datastructures import MultiValueDict
>>> d = MultiValueDict({"k": ["v1", "v2"]})
>>> d.items()
[('k', 'v2')]
>>> super(MultiValueDict, d).items()
[('k', ['v1', 'v2'])]

The original code uses six to cover both Python 2 and 3. This is what Python 2 executes:

def lists(self):
    return list(self.iterlists())

def iterlists(self):
    """Yields (key, list) pairs."""
    return super(MultiValueDict, self).iteritems()

In Python 2, OrderedDict is implemented in pure-Python and relies on self[key], i.e. __getitem__, to retrieve values:

def iteritems(self):
    'od.iteritems -> an iterator over the (key, value) pairs in od'
    for k in self:
        yield (k, self[k])

As such, it picks up the overridden __getitem__ from the MRO and returns only individual values, not the entire lists.

This problem is sidestepped in most builds of Python 3.5+, since OrderedDict usually has a C-implementation available, accidentally shielding its methods from using overridden ones.

collections.OrderedDict is now implemented in C, which makes it 4 to 100 times faster.[What's new in Python 3.5]