a += b not the same as a = a + b [duplicate]

Question

Possible Duplicate:
Why does += behave unexpectedly on lists?

I found an interesting "feature" of the python language today, that gave me much grief.

>>> a = [1, 2, 3]
>>> b = "lol"
>>> a = a + b 
TypeError: can only concatenate list (not "str") to list
>>> a += b
>>> a
[1, 2, 3, 'l', 'o', 'l']

How is that? I thought the two were meant to be equivalent! Even worse, this is the code that I had a hell of a time debugging

>>> a = [1, 2, 3]
>>> b = {'omg': 'noob', 'wtf' : 'bbq'}
>>> a = a + b
TypeError: can only concatenate list (not "dict") to list
>>> a += b
>>> a
[1, 2, 3, 'omg', 'wtf']

WTF! I had lists and dicts within my code, and was wondering how the hell I ended up appending the keys of my dict onto a list without ever calling .keys(). As it turns out, this is how.

I thought the two statements were meant to be equivalent. Even ignoring that, I can kind of understand the way you append strings onto lists (since strings are just character arrays) but dictionaries? Maybe if it appended a list of (key, value) tuples, but grabbing only the keys to add to the list seems completely arbitrary.

Does anyone know the logic behind this?

Answer 1

This is and always has been a problem with mutability in general, and operator overloading specifically. C++ is no better.

The expression a + b computes a new list from the objects bound to a and b, which are not modified. When you assign this back to a, you change the binding of one variable to point to the new value. It is expected that + is symmetrical, so you can't add a dict and a list.

The statement a += b modifies the existing list bound to a. Since it does not change the object identity, the changes are visible to all bindings to the object represented by a. The operator += is obviously not symmetrical, it is equivalent to list.extend, which iterates over the second operand. For dictionaries, this means listing the keys.

Discussion:

If an object doesn't implement +=, then Python will translate it into an equivalent statement using + and =. So the two are sometimes equivalent, depending on the type of the objects involved.

The benefit of a += that mutates the referand (as opposed to the operand value, which is a reference) is that the implementation can be more efficient without a corresponding increase in implementation complexity.

In other languages, you might use more obvious notation. For example, in a hypothetical version of Python with no operator overloading, you might see:

a = concat(a, b)

versus

a.extend(a, b)

The operator notation is really just shorthand for these.

Bonus:

Try it with other iterables too.

>>> a = [1,2,3]
>>> b = "abc"
>>> a + b
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can only concatenate list (not "str") to list
>>> a += b
>>> a
[1, 2, 3, 'a', 'b', 'c']

It's useful to be able to do this, because you can append a generator to a list with += and get the generator contents. It's unfortunate that it breaks compatibility with +, but oh well.

Answer 2

The reason behind this is because the python lists (a in your case) implement the __iadd__ method, which in turns calls the __iter__ method on the passed parameter.

The following code snippet illustrates this better:

class MyDict(dict):
    def __iter__(self):
        print "__iter__ was called"
        return super(MyDict, self).__iter__()


class MyList(list):
    def __iadd__(self, other):
        print "__iadd__ was called"
        return super(MyList, self).__iadd__(other)


a = MyList(['a', 'b', 'c'])
b = MyDict((('d1', 1), ('d2', 2), ('d3', 3)))

a += b

print a

The result is:

__iadd__ was called
__iter__ was called
['a', 'b', 'c', 'd2', 'd3', 'd1']

The python interpreter checks if an object implements the __iadd__ operation (+=) and only if it doesn't it will emulate it by doing a add operation followed by an assignment.