Difference between chain(*iterable) vs chain.from_iterable(iterable)

Question

I have been really fascinated by all the interesting iterators in itertools, but one confusion I have had is the difference between these two functions and why chain.from_iterable exists.

from itertools import chain

def foo(n):
    for i in range(n):
        yield [i, i**2]

chain(*foo(5))

chain.from_iterable(foo(5))

What is the difference between the two functions?

Answer 1

The former can only handle unpackable iterables. The latter can handle iterables that cannot be fully unpacked, such as infinite generators.

Consider

>>> from itertools import chain
>>> def inf():
...     i=0
...     while True:
...         i += 1
...         yield (i, i)
... 
>>> x=inf()
>>> y=chain.from_iterable(x)
>>> z=chain(*x)
<hangs forever>

Furthermore, just the act of unpacking is an eager, up-front-cost activity, so if your iterable has effects you want to evaluate lazily, from_iterable is your best option.

Answer 2

chain(*foo(5)) unpacks the whole generator, packs it into a tuple and processes it then.

chain.from_iterable(foo(5)) queries the generator created from foo(5) value for value.

Try foo(1000000) and watch the memory usage go up and up.

Answer 3

* unpacks the iterator, meaning it iterates the iterator in order to pass its values to the function. chain.from_iterable iterates the iterator one by one lazily.