Correlation between columns in DataFrame

Question

I'm pretty new to pandas, so I guess I'm doing something wrong -

I have a DataFrame:

     a     b 0  0.5  0.75 1  0.5  0.75 2  0.5  0.75 3  0.5  0.75 4  0.5  0.75 

df.corr() gives me:

    a   b a NaN NaN b NaN NaN 

but np.correlate(df["a"], df["b"]) gives: 1.875

Why is that? I want to have the correlation matrix for my DataFrame and thought that corr() does that (at least according to the documentation). Why does it return NaN?

What's the correct way to calculate?

Many thanks!

Answer 1

np.correlate calculates the (unnormalized) cross-correlation between two 1-dimensional sequences:

z[k] = sum_n a[n] * conj(v[n+k]) 

while df.corr (by default) calculates the Pearson correlation coefficient.

The correlation coefficient (if it exists) is always between -1 and 1 inclusive. The cross-correlation is not bounded.

The formulas are somewhat related, but notice that in the cross-correlation formula (above) there is no subtraction of the means, and no division by the standard deviations which is part of the formula for Pearson correlation coefficient.

The fact that the standard deviation of df['a'] and df['b'] is zero is what causes df.corr to be NaN everywhere.

---

From the comment below, it sounds like you are looking for Beta. It is related to Pearson's correlation coefficient, but instead of dividing by the product of standard deviations:

you divide by a variance:

---

You can compute Beta using np.cov

cov = np.cov(a, b) beta = cov[1, 0] / cov[0, 0] 

---

import numpy as np import matplotlib.pyplot as plt np.random.seed(100)   def geometric_brownian_motion(T=1, N=100, mu=0.1, sigma=0.01, S0=20):     """     http://stackoverflow.com/a/13203189/190597 (unutbu)     """     dt = float(T) / N     t = np.linspace(0, T, N)     W = np.random.standard_normal(size=N)     W = np.cumsum(W) * np.sqrt(dt)  # standard brownian motion ###     X = (mu - 0.5 * sigma ** 2) * t + sigma * W     S = S0 * np.exp(X)  # geometric brownian motion ###     return S  N = 10 ** 6 a = geometric_brownian_motion(T=1, mu=0.1, sigma=0.01, N=N) b = geometric_brownian_motion(T=1, mu=0.2, sigma=0.01, N=N)  cov = np.cov(a, b) print(cov) # [[ 0.38234755  0.80525967] #  [ 0.80525967  1.73517501]] beta = cov[1, 0] / cov[0, 0] print(beta) # 2.10609347015  plt.plot(a) plt.plot(b) plt.show() 

The ratio of mus is 2, and beta is ~2.1.

---

And you could also compute it with df.corr, though this is a much more round-about way of doing it (but it is nice to see there is consistency):

import pandas as pd df = pd.DataFrame({'a': a, 'b': b}) beta2 = (df.corr() * df['b'].std() * df['a'].std() / df['a'].var()).ix[0, 1] print(beta2) # 2.10609347015 assert np.allclose(beta, beta2)