How to remove duplicates from list of dicts?

Question

I have a list of dictionaries in python as follows:

[{'category': 'software', 'name': 'irssi', 'version': '1.2.0'},
 {'category': 'software', 'name': 'irssi', 'version': '1.1.2'},
 {'category': 'software', 'name': 'hexchat', 'version': '2.14.2'}]

(parsing some data txt file)

What I wanna do:

If category and name are the same I wanna leave the first appearance of a package entry and remove the rest, so the final output would look like:

[{'category': 'software', 'name': 'irssi', 'version': '1.2.0'},
{'category': 'software', 'name': 'hexchat', 'version': '2.14.2'}]

How should I achieve this? I tried converting the list of dictionaries to a dictionary and then iterate over it with dict.items() but with no luck.

Answer 1

Use a set to keep track of all (category, name) pairs you've already seen:

lst = [
    {'category': 'software', 'name': 'irssi', 'version': '1.2.0'},
    {'category': 'software', 'name': 'irssi', 'version': '1.1.2'},
    {'category': 'software', 'name': 'hexchat', 'version': '2.14.2'}
]

seen = set()
result = []

for dic in lst:
    key = (dic['category'], dic['name'])
    if key in seen:
        continue

    result.append(dic)
    seen.add(key)

print(result)
# output: [{'category': 'software', 'name': 'irssi', 'version': '1.2.0'},
#          {'category': 'software', 'name': 'hexchat', 'version': '2.14.2'}]

This can be generalized into a function:

def keep_first(iterable, key=None):
    if key is None:
        key = lambda x: x

    seen = set()
    for elem in iterable:
        k = key(elem)
        if k in seen:
            continue

        yield elem
        seen.add(k)

>>> list(keep_first(lst, lambda d: (d['category'], d['name'])))
[{'category': 'software', 'name': 'irssi', 'version': '1.2.0'},
 {'category': 'software', 'name': 'hexchat', 'version': '2.14.2'}]

Answer 2

Use itertools.groupby, and take first of each group:

def uniq(lst):
    for _, grp in itertools.groupby(lst, lambda d: (d['category'], d['name'])):
        yield list(grp)[0]
lst = [{'category': 'software', 'name': 'irssi', 'version': '1.2.0'},
       {'category': 'software', 'name': 'irssi', 'version': '1.1.2'},
       {'category': 'software', 'name': 'hexchat', 'version': '2.14.2'}]
print(list(uniq(lst))