Making custom containers work with **kwargs (how does Python expand the args?)

Question

I have a custom container class in Python 2.7, and everything works as expected except if I pass try to expand an instance as **kwargs for a function:

cm = ChainableMap({'a': 1})
cm['b'] = 2
assert cm == {'a': 1, 'b': 2} # Is fine
def check_kwargs(**kwargs):
   assert kwargs == {'a': 1, 'b': 2}
check_kwargs(**cm) # Raises AssertionError

I've overridden __getitem__, __iter__, iterkeys, keys, items, and iteritems, (and __eq__ and __repr__) yet none of them seem to be involved in the expansion as **kwargs, what am I doing wrong?

Edit - The working updated source that now inherits from MutableMapping and adds the missing methods:

from itertools import chain
from collections import MutableMapping

class ChainableMap(MutableMapping):
    """
    A mapping object with a delegation chain similar to JS object prototypes::

        >>> parent = {'a': 1}
        >>> child = ChainableMap(parent)
        >>> child.parent is parent
        True

    Failed lookups delegate up the chain to self.parent::

        >>> 'a' in child
        True
        >>> child['a']
        1

    But modifications will only affect the child::

        >>> child['b'] = 2
        >>> child.keys()
        ['a', 'b']
        >>> parent.keys()
        ['a']
        >>> child['a'] = 10
        >>> parent['a']
        1

    Changes in the parent are also reflected in the child::

        >>> parent['c'] = 3
        >>> sorted(child.keys())
        ['a', 'b', 'c']
        >>> expect = {'a': 10, 'b': 2, 'c': 3}
        >>> assert child == expect, "%s != %s" % (child, expect)

    Unless the child is already masking out a certain key::

        >>> del parent['a']
        >>> parent.keys()
        ['c']
        >>> assert child == expect, "%s != %s" % (child, expect)

    However, this doesn't work::

        >>> def print_sorted(**kwargs):
        ...     for k in sorted(kwargs.keys()):
        ...         print "%r=%r" % (k, kwargs[k])
        >>> child['c'] == 3
        True
        >>> print_sorted(**child)
        'a'=10
        'b'=2
        'c'=3

    """
    __slots__ = ('_', 'parent')

    def __init__(self, parent, **data):
        self.parent = parent
        self._ = data

    def __getitem__(self, key):
        try:
            return self._[key]
        except KeyError:
            return self.parent[key]

    def __iter__(self):
        return self.iterkeys()

    def __setitem__(self, key, val):
        self._[key] = val

    def __delitem__(self, key):
        del self._[key]

    def __len__(self):
        return len(self.keys())

    def keys(self, own=False):
        return list(self.iterkeys(own))

    def items(self, own=False):
        return list(self.iteritems(own))

    def iterkeys(self, own=False):
        if own:
            for k in self._.iterkeys():
                yield k
            return
        yielded = set([])
        for k in chain(self.parent.iterkeys(), self._.iterkeys()):
            if k in yielded:
                continue
            yield k
            yielded.add(k)

    def iteritems(self, own=False):
        for k in self.iterkeys(own):
            yield k, self[k]

    def __eq__(self, other):
        return sorted(self.iteritems()) == sorted(other.iteritems())

    def __repr__(self):
        return dict(self.iteritems()).__repr__()

    def __contains__(self, key):
        return key in self._ or key in self.parent

    def containing(self, key):
        """
        Return the ancestor that directly contains ``key``

        >>> p2 = {'a', 2}
        >>> p1 = ChainableMap(p2)
        >>> c = ChainableMap(p1)
        >>> c.containing('a') is p2
        True
        """
        if key in self._:
            return self
        elif hasattr(self.parent, 'containing'):
            return self.parent.containing(key)
        elif key in self.parent:
            return self.parent

    def get(self, key, default=None):
        """
        >>> c = ChainableMap({'a': 1})
        >>> c.get('a')
        1
        >>> c.get('b', 'default')
        'default'
        """
        if key in self:
            return self[key]
        else:
            return default

    def pushdown(self, top):
        """
        Pushes a new mapping onto the top of the delegation chain:

        >>> parent = {'a': 10}
        >>> child = ChainableMap(parent)
        >>> top = {'a': 'apple', 'b': 'beer', 'c': 'cheese'}
        >>> child.pushdown(top)
        >>> assert child == top

        This creates a new ChainableMap with the contents of ``child`` and makes it
        the new parent (the old parent becomes the grandparent):

        >>> child.parent.parent is parent
        True
        >>> del child['a']
        >>> child['a'] == 10
        True
        """
        old = ChainableMap(self.parent)
        for k, v in self.items(True):
            old[k] = v
            del self[k]
        self.parent = old
        for k, v in top.iteritems():
            self[k] = v

Answer 1

When creating a keyword argument dictionary, the behavior is the same as passing your object into the dict() initializer, which results in the dict {'b': 2} for your cm object:

>>> cm = ChainableMap({'a': 1})
>>> cm['b'] = 2
>>> dict(cm)
{'b': 2}

A more detailed explanation of why this is the case is below, but the summary is that your mapping is converted to a Python dictionary in C code which does some optimization if the argument is itself another dict, by bypassing the Python function calls and inspecting the underlying C object directly.

There are a few ways to approach the solution for this, either make sure that the underlying dict contains everything you want, or stop inheriting from dict (which will require other changes as well, at the very least a __setitem__ method).

edit: It sounds like BrenBarn's suggestion to inherit from collections.MutableMapping instead of dict did the trick.

You could accomplish the first method pretty simply by just adding self.update(parent) to ChainableMap.__init__(), but I'm not sure if that will cause other side effects to the behavior of your class.

Explanation of why dict(cm) gives {'b': 2}:

Check out the following CPython code for the dict object:
http://hg.python.org/releasing/2.7.3/file/7bb96963d067/Objects/dictobject.c#l1522

When dict(cm) is called (and when keyword arguments are unpacked), the PyDict_Merge function is called with cm as the b parameter. Because ChainableMap inherits from dict, the if statement at line 1539 is entered:

if (PyDict_Check(b)) {
    other = (PyDictObject*)b;
    ...

From there on, items from other are added to the new dict that is being created by accessing the C object directly, which bypasses all of the methods that you overwrote.

This means that any items in a ChainableMap instance that are accessed through the parent attribute will not be added to the new dictionary created by dict() or keyword argument unpacking.