Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

python Pandas DataFrame copy(deep=False) vs copy(deep=True) vs '='

Could somebody explain to me a difference between

df2 = df1  df2 = df1.copy()  df3 = df1.copy(deep=False) 

I have tried all options and did as follows:

df1 = pd.DataFrame([1,2,3,4,5]) df2 = df1 df3 = df1.copy() df4 = df1.copy(deep=False) df1 = pd.DataFrame([9,9,9]) 

and returned as follows:

df1: [9,9,9] df2: [1,2,3,4,5] df3: [1,2,3,4,5] df4: [1,2,3,4,5] 

So, I observe no difference in the output between .copy() and .copy(deep=False). Why?

I would expect one of the options '=', copy(), copy(deep=False) to return [9,9,9]

What am I missing please?

like image 755
Dariusz Krynicki Avatar asked Sep 20 '17 16:09

Dariusz Krynicki


People also ask

Is pandas copy a deep copy?

Pandas DataFrame copy() MethodBy default, the copy is a "deep copy" meaning that any changes made in the original DataFrame will NOT be reflected in the copy.

What is deep true in pandas?

When deep=True , data is copied but actual Python objects will not be copied recursively, only the reference to the object.

What is deep copy in DataFrame Python?

A deep copy of a DataFrame or a Series object has its own copy of index and data. It is a process in which the copying process occurs recursively. It means first constructing a new collection object and then recursively populating it with copies of the child objects found in the original.

How do I deep copy a Pandas DataFrame?

To create deep copy of Pandas DataFrame, use df. copy() or df. copy(deep=True) method. To create a shallow copy of Pandas DataFrame, use the df.


2 Answers

If you see the object IDs of the various DataFrames you create, you can clearly see what is happening.

When you write df2 = df1, you are creating a variable named df2, and binding it with an object with id 4541269200. When you write df1 = pd.DataFrame([9,9,9]), you are creating a new object with id 4541271120 and binding it to variable df1, but the object with id 4541269200 which was previously bound to df1 continues to live. If there were no variables bound to that object, it will get garbage collected by Python.

In[33]: import pandas as pd In[34]: df1 = pd.DataFrame([1,2,3,4,5]) In[35]: id(df1) Out[35]: 4541269200  In[36]: df2 = df1 In[37]: id(df2) Out[37]: 4541269200  # Same id as df1  In[38]: df3 = df1.copy() In[39]: id(df3) Out[39]: 4541269584  # New object, new id.  In[40]: df4 = df1.copy(deep=False) In[41]: id(df4) Out[41]: 4541269072  # New object, new id.  In[42]: df1 = pd.DataFrame([9, 9, 9]) In[43]: id(df1) Out[43]: 4541271120  # New object created and bound to name 'df1'.  In[44]: id(df2) Out[44]: 4541269200  # Old object's id not impacted. 

Edit: Added on 7/30/2018

Deep copying doesn't work in pandas and the devs consider putting mutable objects inside a DataFrame as an antipattern. Consider the following:

In[10]: arr1 = [1, 2, 3] In[11]: arr2 = [1, 2, 3, 4] In[12]: df1 = pd.DataFrame([[arr1], [arr2]], columns=['A']) In[13]: df1.applymap(id) Out[13]:              A 0  4515714832 1  4515734952  In[14]: df2 = df1.copy(deep=True) In[15]: df2.applymap(id) Out[15]:              A 0  4515714832 1  4515734952  In[16]: df2.loc[0, 'A'].append(55) In[17]: df2 Out[17]:                 A 0  [1, 2, 3, 55] 1   [1, 2, 3, 4] In[18]: df1 Out[18]:                 A 0  [1, 2, 3, 55] 1   [1, 2, 3, 4] 

df2, if it was a true deep copy should have had new ids for the lists contained within it. As a result, when you modify a list inside df2, it affects the list inside df1 as well, because they are the same objects.

like image 51
Karthik V Avatar answered Sep 22 '22 04:09

Karthik V


Deep copy creates new id's of every object it contains while normal copy only copies the elements from the parent and creates a new id for a variable to which it is copied to.

The reason for none of df2, df3 and df4 displaying [9,9,9] is:

In[33]: import pandas as pd In[34]: df1 = pd.DataFrame([1,2,3,4,5]) In[35]: id(df1) Out[35]: 4541269200  In[36]: df2 = df1 In[37]: id(df2) Out[37]: 4541269200  # Same id as df1  In[38]: df3 = df1.copy() In[39]: id(df3) Out[39]: 4541269584  # New object, new id.  In[40]: df4 = df1.copy(deep=False) In[41]: id(df4) Out[41]: 4541269072  # New object, new id.  In[42]: df1 = pd.DataFrame([9, 9, 9]) In[43]: id(df1) Out[43]: 4541271120  # New object created and bound to name 'df1'. 
like image 30
Aman Agrawal Avatar answered Sep 23 '22 04:09

Aman Agrawal