Why does this Python code (compositing a list extension with a map of itself) make my system freeze up?

Question

When I run

a = ['a']
a.extend(map(lambda x: 'b' + x, a))

it locks my system up until I can do a Ctrl+C if I run it as a Python script from the shell, and running it from the interpreter made me have to hard shutdown my laptop.

However,

a = ['a']
a.extend(list(map(lambda x: 'b' + x, a)))

works fine and gives the expected result.

Why does this happen?

At first, I thought it might be because I was trying to extend a with a map function that ran on a itself, so I wrote:

a = ['a']
tmp = map(lambda x: 'b' + x, a)
a.extend(tmp)

However, that also froze up.

Similarly, this seems to work fine:

a = ['a']
tmp = list(map(lambda x: 'b' + x, a))
a.extend(tmp)

Why is this happening?

I'm doing this on Python 3.4.3.

Answer 1

This is because in Python 3.x map() function returns an iterator, which uses the reference of the list passed to it as the second parameter. So as you are iterating over the map iterator, you are also extending the list, and this keeps on going indefinitely , hence you either get MemoryError or you end up with an infinite loop.

Example to show this behavior -

>>> m = map(lambda x: a.extend(x), a)
>>> m
<map object at 0x021E0E70>
>>> for i,x in enumerate(m):
...     print("Hello")
...
Hello
Hello
.... # lots of Hello
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <lambda>
MemoryError

So when you do - a.extend(map(lambda x: 'b' + x, a)) . You are doing something similar to -

a = ['a']
for x in a:
    a.extend('b'+x)

If you try the above code, you will still get the MemoryError or the infinite loop.

When you do -

a.extend(list(map(lambda x: 'b' + x, a)))

You are using up the iterator by converting it into a list, before you are extending the list a , hence it does not end up in an infinite loop. In this case you are doing something similar to-

a = ['a']
templist = []
for x in a:
    templist.extend('b' + x)
a.extend(templist)

So that is why you do not get the error. Please note above code may not be how python internally runs map , its just something similar.

Answer 2

In python 3, an iterator will be generated from map() function.

When seeing a.extend() function, python will find that you want to extend list a with an iterator related to a and automatically help you to do the iterating.

And here the iteration begins.

Firstly, it's a 'a' in a. The iterator inside map() function gives 'a', a 'ba' is generated from your lambda expression and get appended into list a. a becomes ['a', 'ba'] now.

Then, the iterator inside map() function finds that iterating over a is not giving StopIteration due to a's new pal 'ba''s comming. So the iterator inside map() function gives 'ba' for lambda to process. A 'bba' is generated here and get pushed into a.

That's how a's infinite propagation works.

The following code may help:

a = ['a']
import time
a.extend(map(lambda x: ('b' + x, print(x), time.sleep(1))[0], a))

And it should be trivial to understand why your using of list() to transform the iterator into a static list don't trigger this.

Answer 3

I think Python's object management mechanism is different from C/C++, see this:

a = ['a'] 

for x in a: a.append('b')

if you type in your Python command line, you will encounter a infinite loop, and after you input CTRL+C, and

>>> a

you will get a long list that contains 'a' and 'b', and I think in for loop, the a and the a of a.append('b') are the same object and in the same memory. That's what i think.