Python - Find duplicates in a list of dictionaries and group them

Question

i'm not a programmer and also new to python, i have a list of dicts coming from a json file:

# JSON file (film.json)
[{"year": ["1999"], "director": ["Wachowski"], "film": ["The Matrix"], "price": ["19,00"]},
{"year": ["1994"], "director": ["Tarantino"], "film": ["Pulp Fiction"], "price": ["20,00"]},
{"year": ["2003"], "director": ["Tarantino"], "film": ["Kill Bill vol.1"], "price": ["10,00"]},
{"year": ["2003"], "director": ["Wachowski"], "film": ["The Matrix Reloaded"], "price": ["9,99"]},
{"year": ["1994"], "director": ["Tarantino"], "film": ["Pulp Fyction"], "price": ["15,00"]},
{"year": ["1994"], "director": ["E. de Souza"], "film": ["Street Fighter"], "price": ["2,00"]},
{"year": ["1999"], "director": ["Wachowski"], "film": ["The Matrix"], "price": ["20,00"]},
{"year": ["1982"], "director": ["Ridley Scott"], "film": ["Blade Runner"], "price": ["19,99"]}]

i can import json file with:

import json
json_file = open('film.json')
f = json.load(json_file)

but after that i'm not able to find occurrences in f and separate them in groups by film title. This is what i'm looking for to achieve:

## result grouped by 'film'
#group 1
{"year": ["1999"], "director": ["Wachowski"], "film": ["The Matrix"], "price": ["19,00"]}
{"year": ["1999"], "director": ["Wachowski"], "film": ["The Matrix"], "price": ["20,00"]}
#group 2
{"year": ["1994"], "director": ["Tarantino"], "film": ["Pulp Fiction"], "price": ["20,00"]}
{"year": ["1994"], "director": ["Tarantino"], "film": ["Pulp Fyction"], "price": ["15,00"]}
#group X
 ...

Or better:

new_dict = { 'group1':[[],[],...] , 'group2':[[],[],...] , 'groupX':[...] }

At the moment i'm testing with nested for but without luck..

Thank you.

note: "pulp fyction" is a wanted mistake for future implementation with fuzzy string matching, for now i need only a 'duplicates grouper'

note2: with python 2.x

Answer 1

Because your data is not sorted, use a collections.defaultdict() object to materialize a list for new keys, then key by film title:

from collections import defaultdict

grouped = defaultdict(list)

for film in f:
    grouped[film['film'][0]].append(film)

The film['film'][0] value is used to group the films. You would have to create a canonical version of that key if you wanted to use more sophisticated title grouping.

Demo:

>>> from collections import defaultdict
>>> import json
>>> with open('film.json') as film_file:
...     f = json.load(film_file)
... 
>>> grouped = defaultdict(list)
>>> for film in f:
...     grouped[film['film'][0]].append(film)
... 
>>> grouped
defaultdict(<type 'list'>, {u'Street Fighter': [{u'director': [u'E. de Souza'], u'price': [u'2,00'], u'film': [u'Street Fighter'], u'year': [u'1994']}], u'Pulp Fiction': [{u'director': [u'Tarantino'], u'price': [u'20,00'], u'film': [u'Pulp Fiction'], u'year': [u'1994']}], u'Pulp Fyction': [{u'director': [u'Tarantino'], u'price': [u'15,00'], u'film': [u'Pulp Fyction'], u'year': [u'1994']}], u'The Matrix': [{u'director': [u'Wachowski'], u'price': [u'19,00'], u'film': [u'The Matrix'], u'year': [u'1999']}, {u'director': [u'Wachowski'], u'price': [u'20,00'], u'film': [u'The Matrix'], u'year': [u'1999']}], u'Blade Runner': [{u'director': [u'Ridley Scott'], u'price': [u'19,99'], u'film': [u'Blade Runner'], u'year': [u'1982']}], u'Kill Bill vol.1': [{u'director': [u'Tarantino'], u'price': [u'10,00'], u'film': [u'Kill Bill vol.1'], u'year': [u'2003']}], u'The Matrix Reloaded': [{u'director': [u'Wachowski'], u'price': [u'9,99'], u'film': [u'The Matrix Reloaded'], u'year': [u'2003']}]})
>>> from pprint import pprint
>>> pprint(dict(grouped))
{u'Blade Runner': [{u'director': [u'Ridley Scott'],
                    u'film': [u'Blade Runner'],
                    u'price': [u'19,99'],
                    u'year': [u'1982']}],
 u'Kill Bill vol.1': [{u'director': [u'Tarantino'],
                       u'film': [u'Kill Bill vol.1'],
                       u'price': [u'10,00'],
                       u'year': [u'2003']}],
 u'Pulp Fiction': [{u'director': [u'Tarantino'],
                    u'film': [u'Pulp Fiction'],
                    u'price': [u'20,00'],
                    u'year': [u'1994']}],
 u'Pulp Fyction': [{u'director': [u'Tarantino'],
                    u'film': [u'Pulp Fyction'],
                    u'price': [u'15,00'],
                    u'year': [u'1994']}],
 u'Street Fighter': [{u'director': [u'E. de Souza'],
                      u'film': [u'Street Fighter'],
                      u'price': [u'2,00'],
                      u'year': [u'1994']}],
 u'The Matrix': [{u'director': [u'Wachowski'],
                  u'film': [u'The Matrix'],
                  u'price': [u'19,00'],
                  u'year': [u'1999']},
                 {u'director': [u'Wachowski'],
                  u'film': [u'The Matrix'],
                  u'price': [u'20,00'],
                  u'year': [u'1999']}],
 u'The Matrix Reloaded': [{u'director': [u'Wachowski'],
                           u'film': [u'The Matrix Reloaded'],
                           u'price': [u'9,99'],
                           u'year': [u'2003']}]}

Using SoundEx to group films would be as simple as:

from itertools import groupby, islice, ifilter

_codes = ('bfpv', 'cgjkqsxz', 'dt', 'l', 'mn', 'r')
_sounds = {c: str(i) for i, code in enumerate(_codes, 1) for c in code}
_sounds.update(dict.fromkeys('aeiouy'))
def soundex(word, _sounds=_sounds):
    grouped = groupby(_sounds[c] for c in word.lower() if c in _sounds)
    if _sounds.get(word[0].lower()):
        next(grouped)  # remove first group.
    sdx = ''.join([k for k, g in islice((g for g in grouped if g[0]), 3)])
    return word[0].upper() + format(sdx, '<03')

grouped_by_soundex = defaultdict(list)
for film in f:
    grouped_by_soundex[soundex(film['film'][0])].append(film)

resulting in:

>>> pprint(dict(grouped_by_soundex))
{u'B436': [{u'director': [u'Ridley Scott'],
            u'film': [u'Blade Runner'],
            u'price': [u'19,99'],
            u'year': [u'1982']}],
 u'K414': [{u'director': [u'Tarantino'],
            u'film': [u'Kill Bill vol.1'],
            u'price': [u'10,00'],
            u'year': [u'2003']}],
 u'P412': [{u'director': [u'Tarantino'],
            u'film': [u'Pulp Fiction'],
            u'price': [u'20,00'],
            u'year': [u'1994']},
           {u'director': [u'Tarantino'],
            u'film': [u'Pulp Fyction'],
            u'price': [u'15,00'],
            u'year': [u'1994']}],
 u'S363': [{u'director': [u'E. de Souza'],
            u'film': [u'Street Fighter'],
            u'price': [u'2,00'],
            u'year': [u'1994']}],
 u'T536': [{u'director': [u'Wachowski'],
            u'film': [u'The Matrix'],
            u'price': [u'19,00'],
            u'year': [u'1999']},
           {u'director': [u'Wachowski'],
            u'film': [u'The Matrix Reloaded'],
            u'price': [u'9,99'],
            u'year': [u'2003']},
           {u'director': [u'Wachowski'],
            u'film': [u'The Matrix'],
            u'price': [u'20,00'],
            u'year': [u'1999']}]}