Pandas corr() vs corrwith()

Question

What is the reason of Pandas to provide two different correlation functions?

DataFrame.corrwith(other, axis=0, drop=False): Correlation between rows or columns of two DataFrame objectsCompute pairwise

vs.

DataFrame.corr(method='pearson', min_periods=1): Compute pairwise correlation of columns, excluding NA/null values

(from pandas 0.20.3 documentation)

Answer 1

Basic Answer:

Here's an example that might make it more clear:

np.random.seed(123)
df1=pd.DataFrame( np.random.randn(3,2), columns=list('ab') )
df2=pd.DataFrame( np.random.randn(3,2), columns=list('ac') )

As noted by @ffeast, use corr to compare numerical columns within the same dataframe. Non-numerical columns will automatically be skipped.

df1.corr()

          a         b
a  1.000000 -0.840475
b -0.840475  1.000000

You can compare columns of df1 & df2 with corrwith. Note that only columns with the same names are compared:

df1.corrwith(df2)

a    0.993085
b         NaN
c         NaN

Additional options:

If you want pandas to ignore the column names and just compare the first row of df1 to the first row of df2, then you could rename the columns of df2 to match the columns of df1 like this:

df1.corrwith(df2.set_axis( df1.columns, axis='columns', inplace=False))

a    0.993085
b    0.969220

Note that df1 and df2 need to have the same number of columns in that case.

Finally, a kitchen sink approach: you could also simply horizontally concatenate the two datasets and then use corr(). The advantage is that this basically works regardless of the number of columns and how they are named, but the disadvantage is that you might get more output than you want or need:

pd.concat([df1,df2],axis=1).corr()

          a         b         a         c
a  1.000000 -0.840475  0.993085 -0.681203
b -0.840475  1.000000 -0.771050  0.969220
a  0.993085 -0.771050  1.000000 -0.590545
c -0.681203  0.969220 -0.590545  1.000000

Answer 2

The first one computes correlation with another dataframe:

between rows or columns of two DataFrame objects

The second one computes it with itself

Compute pairwise correlation of columns