Menu
How to remove duplicate items from a list using list comprehension? I have following code:
a = [1, 2, 3, 3, 5, 9, 6, 2, 8, 5, 2, 3, 5, 7, 3, 5, 8]
b = []
b = [item for item in a if item not in b]
but it doesn't work, just produces identical list. Why its producing an identical list?
722
To remove duplicates from a list in Python, iterate through the elements of the list and store the first occurrence of an element in a temporary list while ignoring any other occurrences of that element. The basic approach is implemented in the naive method by: Using a For-loop to traverse the list.
There are multiple ways to remove duplicates other than using Remove Duplicates Stage. As stated above you can use Sort stage, Transformer stage. In sort stage, you can enable Key Change() column and it will be useful to filter the duplicate records. You can use Aggregator stage to remove duplicates.
It's producing an identical list as b contains no elements at run-time.
What you'd want it this:
>>> a = [1, 2, 3, 3, 5, 9, 6, 2, 8, 5, 2, 3, 5, 7, 3, 5, 8]
>>> b = []
>>> [b.append(item) for item in a if item not in b]
[None, None, None, None, None, None, None, None]
>>> b
[1, 2, 3, 5, 9, 6, 8, 7]
>>> a = [10,20,30,20,10,50,60,40,80,50,40,0,100,30,60]
>>> [a.pop(a.index(i, a.index(i)+1)) for i in a if a.count(i) > 1]
>>> print(a)
If you don't mind using a different technique than list comprehension you can use a set for that:
>>> a = [1, 2, 3, 3, 5, 9, 6, 2, 8, 5, 2, 3, 5, 7, 3, 5, 8]
>>> b = list(set(a))
>>> print b
[1, 2, 3, 5, 6, 7, 8, 9]
Use keys on a dict constructed with values in a as its keys.
b = dict([(i, 1) for i in a]).keys()
Or use a set:
b = [i for i in set(a)]
The reason that the list is unchanged is that b starts out empty. This means that if item not in b is always True. Only after the list has been generated is this new non-empty list assigned to the variable b.
4
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
