Add new rows to a pandas dataframe

Question

I have two dataframes df1 and df2 that that were computed from the same source, but with different methods, thus most of the values are same, with some differences. Now, I want to update df1 based on values in df2.

For example:

df1 = pd.DataFrame({'name':['john','deb','john','deb'], 'col1':[490,500,425,678], 'col2':[456,625,578,789],'col3':['TN','OK','OK','NY']})
 name col1 col2 col3
 john  490  456  TN
 deb   500  625  OK
 john  425  578  OK
 deb   678  789  NY

df2 = pd.DataFrame({'name':['deb','john','deb','john','deb'], 'col1':[400,490,500,425,678], 'col2':[225,456,625,578,789],'col3':['TN','TN','OK','OK','NY']})
 name col1 col2 col3
  deb  400  225  TN
 john  490  456  TN
  deb  500  625  OK
 john  425  578  OK
 deb   678  789  NY

So, in this case .append should append only the first row from df2 to df1. So, only if there is a new row in df2 that is not present in df1 (based on name and col3) that column will be added/updated, else it wont be.

This almost seems like something that concat should do.

Answer 1

There are two ways of acheiving your result.

1. Concat both dataframes, then drop duplicates
2. Using an outer join/merge, then drop duplicates

I will show you both.

Concat then Drop

This should be more CPU friendly

df3 = pd.concat([df1,df2])
df3.drop_duplicates(subset=['name', 'col3'], inplace=True, keep='last')

This method is possibly more memory intensive than an outer join because at one point you are holding df1, df2 and the result of the concatination of both [df1, df2] (df3) in memory.

Outer join then Drop

This should be more memory friendly.

df3 = df1.merge(df2, on=list(df1), how='outer')
df3.drop_duplicates(subset=['name', 'col3'], inplace=True, keep='last')

Doing an outer join will make sure you get all entries from both dataframes, but df3 will be smaller than in the case where we use concat.

Version 0.15 and older note:

The keyword keep='last' used to be take_last=True