How do I remove duplicate arrays in a list in Python

Question

I have a list in Python filled with arrays.

([4,1,2],[1,2,3],[4,1,2])

How do I remove the duplicate array?

Answer 1

Very simple way to remove duplicates (if you're okay with converting to tuples/other hashable item) is to use a set as an intermediate element.

lst = ([4,1,2],[1,2,3],[4,1,2])
# convert to tuples
tupled_lst = set(map(tuple, lst))
lst = map(list, tupled_lst)

If you have to preserve order or don't want to convert to tuple, you can use a set to check if you've seen the item before and then iterate through, i.e.,

seen = set()
def unique_generator(lst)
    for item in lst:
       tupled = tuple(item)
       if tupled not in seen:
           seen.add(tupled)
           yield item
lst = list(unique_generator(lst))

This isn't great python, but you can write this as a crazy list comprehension too :)

seen = set()
lst = [item for item in lst if not(tuple(item) in seen or seen.add(tuple(item)))]

Answer 2

If order matters:

>>> from collections import OrderedDict
>>> items = ([4,1,2],[1,2,3],[4,1,2])
>>> OrderedDict((tuple(x), x) for x in items).values()
[[4, 1, 2], [1, 2, 3]]

Else it is much simpler:

>>> set(map(tuple, items))
set([(4, 1, 2), (1, 2, 3)])