Bootstrap t method Python implementation

Question

I have this function to perform a t-test, with samples and populations using the same key in dictionaries. It works well, as intended.

def ttest_(d):
    result = {}
    for k, (l, t) in d.items():
        mean_sample = np.mean(t) 
        mean_population = np.mean(l)
        sd_sample = np.std(t, ddof=1)
        sd_population = np.std(l, ddof=1)
        sample_size = len(t)
        population_size = len(l)
        result[k] = round(((mean_sample - mean_population) /
                                np.sqrt(((sd_sample/np.sqrt(sample_size))**2) +
                                         ((sd_population/np.sqrt(population_size))**2))), 2)

How can I modify this function in order to:

--> Instead of doing the final calculation once, do it it in a bootstrap x times along the lines of something like:

for _ in range(1000)

--> The previous step, would create a distribution of T-tests per key, then the result[k] would be the nth percentile value.... that you could specify with a parameter, and give a value say 0.05.

Edit #1: For clarity, the way I use the function is the following:

pairs = {}
for (k, v), (k2, v2) in product(population.items(), samples.items()):
    if k == k2:
        pairs.update({k: (v, v2)})

Then applied the formula on this dict:

ttest_ = ttest_(pairs)

Edit #2: Is important to preserve this structure of applying the function on a dictionary of dictionaries in order to make the associations between the different samples and keys, and get the associated result[k]. The only difference is adding a bootstrap and the percentile selection.

Edit #3: Thanks to Norman question. To clarify that in the new formula, you compare the same sample[k] with a random sub sample drawn from the population[k], x times, that's how you get the distribution. Those sub-samples are of the size of the original sample[k].

Answer 1

This should do it, if I understood correctly.

from itertools import product
import numpy as np


# Generate fake data.
keys = np.arange(100, 130)
populations = {}
samples = {}
for k in keys:
    loc = np.random.uniform(-9.0, +9.0)
    scale = np.random.uniform(0.4, 4.0)
    n = np.random.randint(400, 800)
    m = np.random.randint(20, 100)
    populations[k] = np.random.normal(loc, scale, n)
    samples[k] = np.random.choice(populations[k], m, replace=False)
    print('data: key={} pop={} samp={}'.format(k, len(populations[k]), len(samples[k])))


def ttest_(d, p, n=1000):
    result = {}
    percentiles = (np.arange(n) + 0.5) / n
    for k, (pop, sample) in d.items():
        size_sample = len(sample)
        mean_sample = np.mean(sample)
        sd_sample = np.std(sample, ddof=1)

        # Generate a distribution of t values.
        tvalues = np.zeros(n)
        for i in range(n):
            sample2 = np.random.choice(pop, size=size_sample, replace=True)
            size_sample2 = len(sample2)
            mean_sample2 = np.mean(sample2)
            sd_sample2 = np.std(sample2, ddof=1)
            # Welch's t-test for sample and sample2.
            tvalues[i] = (mean_sample - mean_sample2) /  \
                         np.sqrt((sd_sample / np.sqrt(size_sample))**2 +
                                 (sd_sample2 / np.sqrt(size_sample2))**2)
        # Interpolate the quantile function at p.
        tvalues.sort()
        result[k] = round(np.interp(p, percentiles, tvalues), 2)
    return result


pairs = {}
for (k, v), (k2, v2) in product(populations.items(), samples.items()):
    if k == k2:
        pairs[k] = (v, v2)

result = ttest_(pairs, p=0.5)
for k, v in result.items():
    print('result: key={} t={}'.format(k, v))