Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Joining Multiple Dataframes with Pandas with overlapping Column Names?

Tags:

merge

join

pandas

I have multiple (more than 2) dataframes I would like to merge. They all share the same value column:

In [431]: [x.head() for x in data]
Out[431]: 
[                     AvgStatisticData
DateTime                             
2012-10-14 14:00:00         39.335996
2012-10-14 15:00:00         40.210110
2012-10-14 16:00:00         48.282816
2012-10-14 17:00:00         40.593039
2012-10-14 18:00:00         40.952014,
                      AvgStatisticData
DateTime                             
2012-10-14 14:00:00         47.854712
2012-10-14 15:00:00         55.041512
2012-10-14 16:00:00         55.488026
2012-10-14 17:00:00         51.688483
2012-10-14 18:00:00         57.916672,
                      AvgStatisticData
DateTime                             
2012-10-14 14:00:00         54.171233
2012-10-14 15:00:00         48.718387
2012-10-14 16:00:00         59.978616
2012-10-14 17:00:00         50.984514
2012-10-14 18:00:00         54.924745,
                      AvgStatisticData
DateTime                             
2012-10-14 14:00:00         65.813114
2012-10-14 15:00:00         71.397868
2012-10-14 16:00:00         76.213973
2012-10-14 17:00:00         72.729002
2012-10-14 18:00:00         73.196415,
....etc

I read that join can handle multiple dataframes, however I get:

In [432]: data[0].join(data[1:])
...
Exception: Indexes have overlapping values: ['AvgStatisticData']

I have tried passing rsuffix=["%i" % (i) for i in range(len(data))] to join and still get the same error. I can workaround this by building my data list in a way where the column names don't overlap, but maybe there is a better way?

like image 527
Kyle Brandt Avatar asked Oct 22 '12 00:10

Kyle Brandt


People also ask

How can I join two DataFrames in Pandas with different column names?

It is possible to join the different columns is using concat() method. DataFrame: It is dataframe name. axis: 0 refers to the row axis and1 refers the column axis. join: Type of join.


1 Answers

In [65]: pd.concat(data, axis=1)
Out[65]:
                     AvgStatisticData  AvgStatisticData  AvgStatisticData  AvgStatisticData
2012-10-14 14:00:00         39.335996         47.854712         54.171233         65.813114
2012-10-14 15:00:00         40.210110         55.041512         48.718387         71.397868
2012-10-14 16:00:00         48.282816         55.488026         59.978616         76.213973
2012-10-14 17:00:00         40.593039         51.688483         50.984514         72.729002
2012-10-14 18:00:00         40.952014         57.916672         54.924745         73.196415
like image 127
Wouter Overmeire Avatar answered Oct 11 '22 17:10

Wouter Overmeire