Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to remove duplicate dictionary based on selected keys from a list of dictionaries in Python?

I am new to Python and trying to learn it as much as possible. I am stuck with a silly problem where I want to remove certain dictionary items of a list based on selective key-value pairs. For ex, I have:

l = [{'A':1, 'B':2, 'C':3, 'D':4},
     {'A':5, 'B':6, 'C':7, 'D':8},
     {'A':1, 'B':9, 'C':3, 'D':10}]

And the output I want is removal of dictionaries based on two keys A and C values:

l = [{'A':1, 'B':2, 'C':3, 'D':4},
     {'A':5, 'B':6, 'C':7, 'D':8}]
like image 523
Mohit Sharma Avatar asked Jan 17 '16 03:01

Mohit Sharma


People also ask

How do I remove duplicates from a dictionary list?

The strategy is to convert the list of dictionaries to a list of tuples where the tuples contain the items of the dictionary. Since the tuples can be hashed, you can remove duplicates using set (using a set comprehension here, older python alternative would be set(tuple(d.


1 Answers

Using set to remember whether the items are seen.

>>> A, B, C, D = 'ABCD'
>>>
>>> lst = [
...     {A:1, B:2, C:3, D:4},
...     {A:5, B:6, C:7, D:8},
...     {A:1, B:9, C:3, D:10}
... ]
>>> seen = set()
>>> [x for x in lst if [(x[A], x[C]) not in seen, seen.add((x[A], x[C]))][0]]
[{'A': 1, 'C': 3, 'B': 2, 'D': 4}, {'A': 5, 'C': 7, 'B': 6, 'D': 8}]
like image 106
falsetru Avatar answered Sep 28 '22 10:09

falsetru