Why doesn't python take advantage of __iadd__ for sum and chained operators?

Question

I just conducted an interesting test:

~$ python3 # I also conducted this on python 2.7.6, with the same result
Python 3.4.0 (default, Apr 11 2014, 13:05:11) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> class Foo(object):
...     def __add__(self, other):
...         global add_calls
...         add_calls += 1
...         return Foo()
...     def __iadd__(self, other):
...         return self
...
>>> add_calls = 0
>>> a = list(map(lambda x:Foo(), range(6)))
>>> a[0] + a[1] + a[2]
<__main__.Foo object at 0x7fb588e6c400>
>>> add_calls
2
>>> add_calls = 0
>>> sum(a, Foo())
<__main__.Foo object at 0x7fb588e6c4a8>
>>> add_calls
6

Obviously, the __iadd__ method is more efficient than the __add__ method, not requiring the allocation of a new class. If my objects being added were sufficiently complicated, this would create unnecessary new objects, potentially creating huge bottlenecks in my code.

I would expect that, in an a[0] + a[1] + a[2], the first operation would call __add__, and the second operation would call __iadd__ on the newly created object.

Why doesn't python optimize this?

Answer 1

The __add__ method is free to return a different type of object, while __iadd__ should, if using in-place semantics, return self. They are not required to return the same type of object here, so sum() should not rely on the special semantics of __iadd__.

You can use the functools.reduce() function to implement your desired functionality yourself:

from functools import reduce

sum_with_inplace_semantics = reduce(Foo.__iadd__, a, Foo())

Demo:

>>> from functools import reduce
>>> class Foo(object):
...     def __add__(self, other):
...         global add_calls
...         add_calls += 1
...         return Foo()
...     def __iadd__(self, other):
...         global iadd_calls
...         iadd_calls += 1
...         return self
... 
>>> a = [Foo() for _ in range(6)]
>>> result = Foo()
>>> add_calls = iadd_calls = 0
>>> reduce(Foo.__iadd__, a, result) is result
True
>>> add_calls, iadd_calls
(0, 6)