Python: inverse empirical cumulative distribution function (ECDF)?

Question

We can create the ECDF with

import numpy as np
from statsmodels.distributions.empirical_distribution import ECDF
ecdf = ECDF([3, 3, 1, 4])

and obtain then ECDF at point with

ecdf(x)

However, what if I want to know the x for percentile 97.5% ?

From http://www.statsmodels.org/stable/generated/statsmodels.distributions.empirical_distribution.ECDF.html?highlight=ecdf, it seems like not been implemented.

Is there any way to do this? Or any other libraries?

Answer 1

Since the empirical CDF just places mass of 1/n at each data point, the 97.5th quantile is just the data point that is bigger than 97.5% of all the other points. To find this value, you can simply sort the data in ascending order and find the 0.975n-th largest value.

sample = [1, 5, 2, 10, -19, 4, 7, 2, 0, -1]
n = len(sample)
sort = sorted(sample)
print sort[int(n * 0.975)]

Which produces:

10

Since we remember than for discrete distributions (like the empirical cdf), the quantile function is defined here , we realize that we have to take the 0.975n-th (rounded up) largest value.

Answer 2

This is my suggestion. Linear interpolation because dfs are only effectively estimated from fairly large samples anyway. The interpolating line segments can be obtained because their endpoints occur at distinct values in the sample.

import statsmodels.distributions.empirical_distribution as edf
from scipy.interpolate import interp1d
import numpy as np
import matplotlib.pyplot as plt

sample = [1,4,2,6,5,5,3,3,5,7]
sample_edf = edf.ECDF(sample)

slope_changes = sorted(set(sample))

sample_edf_values_at_slope_changes = [ sample_edf(item) for item in slope_changes]
inverted_edf = interp1d(sample_edf_values_at_slope_changes, slope_changes)

x = np.linspace(0.1, 1)
y = inverted_edf(x)
plt.plot(x, y, 'ro', x, y, 'b-')
plt.show()

print ('97.5 percentile:', inverted_edf(0.975))

It produces the following output,

97.5 percentile: 6.75

and this graph.

Answer 3

numpy.quantile(x, q=.975) will return the value along array x at which has ecdf 0.975.

Similarly, there is pandas.quantile(q=0.97) for Series/DataFrames.