Python function to merge unique values form multiple lists to one list

Question

I am pretty new to Python. I am trying to write a function that will merge unique values in separate lists into one list. I keep getting a result of a tuple of lists. Ultimately I would like to have one list of unique values from my three lists -a,b,c. Can anyone give me a hand with this?

def merge(*lists):
    newlist = lists[:]
    for x in lists:
        if x not in newlist:
            newlist.extend(x)
    return newlist

a = [1,2,3,4]
b = [3,4,5,6]
c = [5,6,7,8]

print(merge(a,b,c))

I am getting a Tuple of Lists

([1, 2, 3, 4], [3, 4, 5, 6], [5, 6, 7, 8])

Answer 1

You may just need sets:

>>> a = [1,2,3,4]
>>> b = [3,4,5,6]
>>> c = [5,6,7,8]
>>>
>>> uniques = set( a + b + c )
>>> uniques
set([1, 2, 3, 4, 5, 6, 7, 8])
>>>

Answer 2

If you don't care about them being in the original order, the easiest and likely fasted way is to use set functions:

>>> set().union(a, b, c)
{1, 2, 3, 4, 5, 6, 7, 8}

If you do care about the original order (sets happen to preserve it in this case, but aren't guaranteed to), then you can fix your original attempt by realising that the argument lists contains a tuple of all of the original lists you passed in. This means that iterating over it gets you each of those lists one at a time, rather than the elements in them - you can fix this by using the itertools module:

for x in itertools.chain.from_iterable(lists):
   if x not in newlist:
      newlist.append(x)

Also, you would want newlist to start out as an empty list rather than a copy of the input lists.

Answer 3

Handling a Dynamically Generated List of Lists

A common use case is generating a list of lists dynamically, with each child list sometimes having an arbitrary length:

import random
abc, values = [], ["a", "b", "c", "d"]
for i in range(3):
    l = []
    for j in range(3):
        l.append(values[random.randint(0, len(values) - 1)])
    abc.append(l)

If you're working with a list of lists, simply summing them as proposed by g.d.d.c. doesn't work. That is:

uniques = set( a + b + c )

The hiccup comes from the fact that you have to specifically reference the lists a, b, and c. The answer by Ivc is really nice and gets us closer:

set().union(a, b, c)

But again, you have to explicitly reference your lists.

The Solution

To get the unique values from a list of lists with arbitrary lengths, you can use positional expansion:

import random
abc, values = [], ["a", "b", "c", "d"]
for i in range(3):
    l = []
    for j in range(3):
        l.append(values[random.randint(0, len(values) - 1)])
    abc.append(l)
# The Important Line Below
unique = set().union(*abc)
print(unique)

Which will return the appropriate values unordered (e.g. ["d", "b", "a", "d"])