How to merge N Python dictionaries without overwriting values?

Question

I have this list of dictionaries:

list_of_ds = [
    {'a': [1, 2], 'b': [4, 5], 'c': [6, 7]},
    {'a': [4], 'b': [56], 'c': [46]},
    {'a': [92], 'b': [65], 'c': [43]}
]

and I want this as output:

{'a': [1, 2, 4, 92], 'b': [4, 5, 56, 65], 'c': [6, 7, 46, 43]}

Until now ...

I tried

d_of_ds = reduce(lambda d1, d2: d1.update(d2), list_of_ds)

gives: AttributeError: 'NoneType' object has no attribute 'update'

I tried

d_of_ds = reduce(lambda d1, d2: d1.update(d2) or d1, list_of_ds, {})

Overwrites each iteration: {'a': [92], 'b': [65], 'c': [43]}

I tried

d_of_ds = {k: v for d in list_of_ds for k, v in d.items()}

Overwrites each iteration: {'a': [92], 'b': [65], 'c': [43]}

Answer 1

Using reduce is not a good idea in this case.Also your lambda function has a wrong logic at all, because you are trying to update the whole of dictionaries together, not their elements see the following :

>>> a={'a':[1,2], 'b':[4,5],'c':[6,7]}
>>> a.update({'a':[4], 'b':[56],'c':[46]})
>>> a
{'a': [4], 'c': [46], 'b': [56]}

But as a more efficient way you can use dict.setdefault method :

>>> new={}
>>> for d in list_of_ds:
...    for i,j in d.items():
...       new.setdefault(i,[]).extend(j)
... 
>>> new
{'a': [1, 2, 4, 92], 'c': [6, 7, 46, 43], 'b': [4, 5, 56, 65]}

Also you can use collections.defaultdict :

>>> from collections import defaultdict
>>> d=defaultdict(list)
>>> for sub in list_of_ds:
...    for i,j in sub.items():
...       d[i].extend(j)
... 
>>> d
defaultdict(<type 'list'>, {'a': [1, 2, 4, 92], 'c': [6, 7, 46, 43], 'b': [4, 5, 56, 65]})

Answer 2

Not-so-efficient but pretty reduce solution:

def f(x, y):
    return {k: x.get(k, []) + y.get(k, []) for k in set(x).union(y)}

from functools import reduce
reduce(f, list_of_ds) # {'b': [4, 5, 56, 65], 'a': [1, 2, 4, 92], 'c': [6, 7, 46, 43]}

Not-so-pretty but efficient solution using collections.defaultdict:

from collections import defaultdict

def f(list_of_ds):
    new = defaultdict(list)
    for d in list_of_ds:
        for k, v in d.items():
            new[k].extend(v)
    return new # or dict(new), to obtain a 'dict' object

f(list_of_ds) # {'a': [1, 2, 4, 92], 'b': [4, 5, 56, 65], 'c': [6, 7, 46, 43]}