Nested generator expressions behave unexpectedly

Question

With the following code:

A = [1, 2] B = [-2, -1] C = [-1, 2] D = [0, 2]  ab = (a + b for a in A for b in B) cd = (c + d for c in C for d in D) abcd = (e_ab + e_cd for e_ab in ab for e_cd in cd) 

The len(abcd) is expected to be 16, but it is actually 4. If I used a list comprehension instead, the problem goes away. Why is that?

Answer 1

You can only ride the generator train once, after it reaches its destination, no more rides. In your case, the cd generator is exhausted and then can't be iterated through again.

list objects, on the other hand, create a separate iterator object every time you call iter on them (which the for loop implicitly does for you):

print(iter([1, 2, 3])) # <list_iterator at 0x7f18495d4c88>  

and produce a fresh iterator you can use. This happens any time iter is invoked on it; since a new object is produced every time, you can go through lists multiple times. Multiple rides!

In short, if you only change cd to be a list (in general, the object that will be iterated through multiple times):

ab = (a + b for a in A for b in B) cd = [c + d for c in C for d in D]  # list-comp instead 

it will yield the wanted result by creating fresh iterator objects from cd for every element in ab:

abcd = (e_ab + e_cd for e_ab in ab for e_cd in cd) print(len(list(abcd))) # 16 

of course you can achieve this by using product from itertools too but, that's beyond the point of why this happens.

Answer 2

I think this is because you can only iterate over generator once. So after you looped thorough e_cd first time this will not produced anything on another iteration of external cycle.