Count values in overlapping sliding windows in python

Question

Given an array, a, of sorted values, and an array of ranges, bins, what is the most efficient way to count how many values in a fall within each range, rng, in bins?

Currently I am doing the following:

def sliding_count(a, end, window, start=0, step=1):
    bins = [(x, x + window) for x in range(start, (end + 1) - window, step)]
    counts = np.zeros(len(bins))
    for i, rng in enumerate(bins):
        count = len(a[np.where(np.logical_and(a>=rng[0], a<=rng[1]))])
        counts[i] = count
    return counts

a = np.array([1, 5, 8, 11, 14, 19])
end = 20
window = 10
sliding_count(a, end, window)

Which returns the expected array

array([3., 4., 3., 3., 4., 4., 3., 3., 3., 3., 3.])

But I feel like there must be a more effective way of doing this?

Answer 1

import numpy as np

def alt(a, end, window, start=0, step=1):
    bin_starts = np.arange(start, end+1-window, step)
    bin_ends = bin_starts + window
    last_index = np.searchsorted(a, bin_ends, side='right')
    first_index = np.searchsorted(a, bin_starts, side='left')
    return  last_index - first_index

def sliding_count(a, end, window, start=0, step=1):
    bins = [(x, x + window) for x in range(start, (end + 1) - window, step)]
    counts = np.zeros(len(bins))
    for i, rng in enumerate(bins):
        count = len(a[np.where(np.logical_and(a>=rng[0], a<=rng[1]))])
        counts[i] = count
    return counts

a = np.array([1, 5, 8, 11, 14, 19])
end = 20
window = 10

print(sliding_count(a, end, window))
# [3. 4. 3. 3. 4. 4. 3. 3. 3. 3. 3.]

print(alt(a, end, window))
# [3 4 3 3 4 4 3 3 3 3 3]

---

How alt works:

Generate the starting and ending values of the bins:

In [73]: bin_starts = np.arange(start, end+1-window, step); bin_starts
Out[73]: array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10])

In [74]: bin_ends = bin_starts + window; bin_ends
Out[74]: array([10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20])

Since a is in sorted order, you can use np.searchsorted to find the first and last index in bin_starts and bin_ends where each value in a fits:

In [75]: last_index = np.searchsorted(a, bin_ends, side='right'); last_index
Out[75]: array([3, 4, 4, 4, 5, 5, 5, 5, 5, 6, 6])

In [76]: first_index = np.searchsorted(a, bin_starts, side='left'); first_index
Out[76]: array([0, 0, 1, 1, 1, 1, 2, 2, 2, 3, 3])

The count is simply the difference in indices:

In [77]: last_index - first_index
Out[77]: array([3, 4, 3, 3, 4, 4, 3, 3, 3, 3, 3])

---

Here is a perfplot comparing the performance of alt versus sliding_count as a function of the length of a:

import perfplot

def make_array(N):
    a = np.random.randint(10, size=N)
    a = a.cumsum()
    return a

def using_sliding(a):
    return sliding_count(a, end, window)

def using_alt(a):
    return alt(a, end, window)

perfplot.show(
    setup=make_array,
    kernels=[using_sliding, using_alt],
    n_range=[2**k for k in range(22)],
    logx=True,
    logy=True,
    xlabel='len(a)')

Perfplot also checks that the value returned by using_sliding equals the value returned by using_alt.

Matt Timmermans' idea, "subtract position_in_a from the count for that bin" triggered this solution.