Combine lists into a single iterable that can be reused

Question

I have an issue where I have some number of lists (2 or more) that I need to combine into one iterable. The problem is, the lists that make up the iterable at constantly updating and I want my iterable to get those changes without any extra work.

Just combining lists don't work, because adding a list to a list makes a new list.

list_a = ['foo', 'bar']
list_b = ['abc']
list_c = list_a + list_b
list_a.append('ttt')
print(list_c)
# Result: ['foo', 'bar', 'abc']

I could add the lists into a list to make a list of lists, and that'd work, but unpacking this will complicate my production-code's logic too much to be viable.

list_a = ['foo', 'bar']
list_b = ['abc']
list_c = [list_a, list_b]
list_a.append('ttt')
print(list_c)
# Result: [['foo', 'bar', 'ttt'], ['abc']]

I like the idea of itertools.chain because it gets me part of the way there but I can only iterate over the lists once before I lose my references to the original list.

import itertools

list_a = ['foo', 'bar']
list_b = ['abc']
iter_c = itertools.chain(list_a, list_b)
list_a.append('ttt')

for item in iter_c:
    print(item)  # Works fine here

for item in iter_c:
    print(item)  # The iterable was exhausted - this doesn't work anymore

You can cast the chain to a list but modifications for list_a aren't going to carry over once you do.

import itertools

list_a = ['foo', 'bar']
list_b = ['abc']
iter_c = itertools.chain(list_a, list_b)
list_a.append('ttt')
list_c = list(iter_c)

for item in list_c:
    print(item)  # Works fine here

for item in list_c:
    print(item)  # Now works fine here, too

list_a.append('zzz')  # This won't get added to our chain
print(list_c)
# Result: ['foo', 'bar', 'ttt', 'abc']

I built a hacky class to do what I want but I'm very unsatisfied with it.

import collections
import uuid


class IterGroup(object):
    def __init__(self):
        super(IterGroup, self).__init__()
        self._data = collections.OrderedDict()

    def append(self, item):
        # The key doesn't matter as long as it's unique. The key is ignored.
        self._data[str(uuid.uuid4())] = item

    def __iter__(self):
        for items in self._data.values():
            for item in items:
                yield item


list_a = ['foo', 'bar']
list_b = ['abc']
list_c = IterGroup()  # Not really a list but just go with it
list_c.append(list_a)
list_c.append(list_b)

list_a.append('ttt')

for item in list_c:
    print(item)

list_a.append('zzz')

for item in list_c:
    print(item)
# Prints ['foo', 'bar', 'ttt', 'zzz', 'abc']

So my criteria for the desired solution is

1. Must be able to add iterables to it
2. Mutating one of the iterables after it's been added will be automatically reflected in the group-solution
3. The combined iterable can be iterated as many times as we'd like
4. Must maintain order
5. Ideally, I could also get items by-index, though not required
6. I'd rather the solution not be a custom class (but I won't be choosy if that's how it has to be)
7. Must work in Python 2

Has anyone got a clean solution to this problem?

Answer 1

I don' think there's a way around using a custom class for this behaviour, but I don't see why that class should extend OrderedDict; just implement the __iter__ and __getitem__ methods. You can try something like this.

class Multilistview:

    def __init__(self, *lists):
        self.lists = lists

    def __iter__(self):
        return itertools.chain.from_iterable(self.lists)

    def __getitem__(self, idx):
        if isinstance(idx, slice):
            return list(itertools.islice(self, idx.start, idx.stop, idx.step))
        else:
            for i, x in enumerate(self):
                if i == idx:
                    return x

Or simpler, but this will materialize the entire list each time you ask for an item:

    def __getitem__(self, idx):
        return list(self)[idx]

(You could also make __getitem__ much more efficient by checking the length of each item to determine which list to use and the "corrected" index in that list.)

Example

list_a = ['foo', 'bar']
list_b = ['abc']
list_c = Multilistview(list_a, list_b)
for x in list_c:
    print(x)
# foo
# bar
# abc
list_a.append('blub')
list_b[:] = [1,2,3]
print(list(list_c))
# ['foo', 'bar', 'blub', 1, 2, 3]
print(list_c[4])
# 2
print(list_c[2:5])
# ['blub', 1, 2]

Answer 2

Specific - two lists:

def f(l1, l2): return lambda: l1 + l2

L1 = [1, 2, 3]
L2 = [4, 5]
newl = f(L1, L2)
print(newl())

L2.append(6)
print(newl())

More general - any number of lists:

def f(*L):
  def g():
    out = []
    for li in L:
      out += li
    return out
  return g

L1 = [1,2]
L2 = [3,4]
L3 = [5]
newl = f(L1,L2,L3)
print(newl())

L3.append(6)
print(newl())

Even more general - any mutable iterator, any combination:

def f(combine, *iters): return lambda: combine(*iters)

def makelist(*L):
  out = []
  for li in L:
    out += li
  return out

def makegenerator(*L):
  for li in L:
    for i in li:
      yield i

def makedict(*D):
  out = dict()
  for di in D:
    for key in di:
      out[key] = di.get(key)
  return out

def ListsToList(*L): return f(makelist, *L)

def ListsToGenerator(*L): return f(makegenerator, *L)

def DictsToDict(*D): return f(makedict, *D)

L1 = [1,2]
L2 = [3,4]
L3 = [5]
newl = ListsToList(L1,L2,L3)
newg = ListsToGenerator(L1,L2,L3)

L3.append(7)
print(newl())
print([i for i in newg()])

D1 = {"x": 1, "y": 2}
D2 = {"a": 5, "b": 10}
newd = DictsToDict(D1,D2)

D2["c"] = 15
print(newd())