Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Intersection of multiple pandas dataframes

I have a number of dataframes (100) in a list as:

frameList = [df1,df2,..,df100]

Each dataframe has the two columns DateTime, Temperature.

I want to intersect all the dataframes on the common DateTime column and get all their Temperature columns combined/merged into one big dataframe: Temperature from df1, Temperature from df2, Temperature from df3, .., Temperature from df100.

(pandas merge doesn't work as I'd have to compute multiple (99) pairwise intersections).

like image 919
Zanam Avatar asked Nov 10 '16 17:11

Zanam


1 Answers

Use pd.concat, which works on a list of DataFrames or Series.

pd.concat(frameList, axis=1, join='inner')

This is better than using pd.merge, as pd.merge will copy the data pairwise every time it is executed. pd.concat copies only once. However, pd.concat only merges based on an axes, whereas pd.merge can also merge on (multiple) columns.

like image 103
Olaf Avatar answered Sep 20 '22 21:09

Olaf