functools reduce In-Place modifies original dataframe

Question

I currently facing the issue that "functools.reduce(operator.iadd,...)" alters the original input. E.g.

I have a simple dataframe df = pd.DataFrame([[['A', 'B']], [['C', 'D']]])

        0
0  [A, B]
1  [C, D]

Applying the iadd operator leads to following result:

functools.reduce(operator.iadd, df[0])
['A', 'B', 'C', 'D']

Now, the original df changed to

              0
0  [A, B, C, D]
1        [C, D]

Also copying the df using df.copy(deep=True) beforehand does not help.

Has anyone an idea to overcome this issue? THX, Lazloo

Answer 1

Use operator.add instead of operator.iadd:

In [8]: functools.reduce(operator.add, df[0])
Out[8]: ['A', 'B', 'C', 'D']

In [9]: df
Out[9]: 
        0
0  [A, B]
1  [C, D]

After all, operator.iadd(a, b) is the same as a += b. So it modifies df[0]. In contrast, operator.add(a, b) returns a + b, so there is no modification of df[0].

---

Or, you could compute the same quantity using df[0].sum():

In [39]: df[0].sum()
Out[39]: ['A', 'B', 'C', 'D']

---

The docs for df.copy warns:

When deep=True, data is copied but actual Python objects will not be copied recursively, only the reference to the object.

Since df[0] contains Python lists, the lists are not copied even with df.copy(deep=True). This is why modifying the copy still affects df.