Python3 How to join two lists of dicts by unique keys

Question

I have two lists:

list1 = [ {'sth': 13, 'important_key1': 'AA', 'important_key2': '3'}, {'oh!': 14, 'important_key1': 'FF', 'important_key2': '4'}, {'sth_else': 'abc', 'important_key1': 'ZZ', 'important_key2': '5'}]
list2 = [ {'why-not': 'tAk', 'important_key1': 'GG', 'important_key2': '4'}, {'hmmm': 'no', 'important_key1': 'AA', 'important_key2': '3'}]

I want to return a list with objects only from list1 but if the same important_key1 and important_key2 is in any element in list2 I want this element from list2.

So the output should be:

[ {'hmmm': 'no', 'important_key1': 'AA', 'important_key2': '3'}, {'oh!': 14, 'important_key1': 'FF', 'important_key2': '4'}, {'sth_else': 'abc', 'important_key1': 'ZZ', 'important_key2': '5'}]

It is not complicated to do it by two or three loops but I wonder whether there is a simple way by using list comprehensions or something like that.

This is the "normal" way:

list1 = [ {'sth': 13, 'important_key1': 'AA', 'important_key2': '3'}, {'oh!': 14, 'important_key1': 'FF', 'important_key2': '4'}]
list2 = [ {'hmmm': 'no', 'important_key1': 'AA', 'important_key2': '3'}, {'why-not': 'tAk', 'important_key1': 'GG', 'important_key2': '4'}]

final_list = []
for element in list1:
    there_was_in_list2 = False
    for another_element in list2:
        if element['important_key1'] == another_element['important_key1'] and element['important_key2'] == another_element['important_key2']:
            final_list.append(another_element)
            there_was_in_list2 = True
            break
    if not there_was_in_list2:
        final_list.append(element)
print(final_list)

is there any Pythonic way to do that?

Answer 1

You can use a list comprehension:

list1 = [{'sth': 13, 'important_key1': 'AA', 'important_key2': '3'}, {'oh!': 14, 'important_key1': 'FF', 'important_key2': '4'}]
list2 = [{'hmmm': 'no', 'important_key1': 'AA', 'important_key2': '3'}, {'why-not': 'tAk', 'important_key1': 'GG', 'important_key2': '4'}]
vals = ['important_key1', 'important_key2']
new_list = [[c if any(c[a] == i[a] for a in vals) else i for c in list2] for i in list1]
final_result = [i[0] for i in new_list if i]

Output:

[{'hmmm': 'no', 'important_key1': 'AA', 'important_key2': '3'}, {'oh!': 14, 'important_key1': 'FF', 'important_key2': '4'}]

Answer 2

You can convert list2 to a dict indexed by a tuple of the values the important keys in list2, and then use it to determine if the same keys in list1 have the same values as you iterate through list1 in a list comprehension, so that the time complexity gets reduced to O(n) from your O(n*m):

keys = ['important_key1', 'important_key2']
d2 = {tuple(d[k] for k in keys): d for d in list2[::-1]}
print([d2.get(tuple(d[k] for k in keys), d) for d in list1])

This outputs (with your sample input):

[{'hmmm': 'no', 'important_key1': 'AA', 'important_key2': '3'}, {'oh!': 14, 'important_key1': 'FF', 'important_key2': '4'}, {'sth_else': 'abc', 'important_key1': 'ZZ', 'important_key2': '5'}]

As you described in your question, only {'sth': 13, 'important_key1': 'AA', 'important_key2': '3'} in list1 would get replaced by {'hmmm': 'no', 'important_key1': 'AA', 'important_key2': '3'} because only this dict has both important_key1 and important_key2 matching those of a dict in list2.