how to convert a dataframe of counts to a probability density function

Question

Suppose that I have the following observations of integers:

df = pd.DataFrame({'observed_scores': [100, 100, 90, 85, 100, ...]})

I know that this can be used as an input to make a density plot:

df['observed_scores'].plot.density()

but suppose that what I have is a counts table:

df = pd.DataFrame({'observed_scores': [100, 95, 90, 85, ...], 'counts': [1534, 1399, 3421, 8764, ...})

which is cheaper to store than the full observed_scores Series (I have LOTS of observations).

I know it's possible to plot the histogram using the counts, but how do I plot the density plot? If possible, can it be done without having to unstack/unravel the counts table into thousands of rows?

Answer 1

IIUC, statsmodels lets you fit a weighted KDE:

from statsmodels.nonparametric.kde import KDEUnivariate

df = pd.DataFrame({'observed_scores': [100, 95, 90, 85],
                   'counts': [1534, 1399, 3421, 8764]})

kde1= KDEUnivariate(df.observed_scores)
kde_noweight = KDEUnivariate(df.observed_scores)
kde1.fit(weights=df.counts, fft=False)
kde_noweight.fit()
plt.plot(kde1.support, kde1.density)
plt.plot(kde_noweight.support, kde_noweight.density)
plt.legend(['weighted', 'unweighted'])

Output: