Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

how to merge two pandas DataFrames and aggregate one specific column

Tags:

python

pandas

I have 2 DataFrames:

         city  count    school
0    New York      1  school_3
1  Washington      1  School_4
2  Washington      1  School_5
3          LA      1  School_1
4          LA      1  School_4

         city  count    school
0    New York      1  School_3
1  Washington      1  School_1
2          LA      1  School_3
3          LA      2  School_4

I want to get the this result:

         city  count    school
0    New York      2  school_3
1  Washington      1  School_1
2  Washington      1  School_4
3  Washington      1  School_5
4          LA      1  School_1
5          LA      1  School_3
6          LA      3  School_4

Following is the code.

d1 = [{'city':'New York', 'school':'school_3', 'count':1},
      {'city':'Washington', 'school':'School_4', 'count':1},
      {'city':'Washington', 'school':'School_5', 'count':1},
      {'city':'LA', 'school':'School_1', 'count':1},
      {'city':'LA', 'school':'School_4', 'count':1}]


d2 = [{'city':'New York', 'school':'School_3', 'count':1},
      {'city':'Washington', 'school':'School_1', 'count':1},
      {'city':'LA', 'school':'School_3', 'count':1},
      {'city':'LA', 'school':'School_4', 'count':2}]

x1 = pd.DataFrame(d1)
x2 = pd.DataFrame(d2)
#just get empty DataFrame
print pd.merge(x1, x2)

How to get the aggregate result ?

like image 505
wangke1020 Avatar asked Jan 26 '15 01:01

wangke1020


1 Answers

You can do:

>>> pd.concat([x1, x2]).groupby(["city", "school"], as_index=False)["count"].sum()
       city    school        count
0          LA  School_1      1
1          LA  School_3      1
2          LA  School_4      3
3    New York  School_3      1
4    New York  school_3      1
5  Washington  School_1      1
6  Washington  School_4      1
7  Washington  School_5      1

Note that New York appears 2 times because of a typo in the data (school_3 vs School_3).

like image 55
elyase Avatar answered Oct 06 '22 00:10

elyase