single iteration sharing the iterator

Question

I have a lot of data, usually in a file. I want to compute some quantities so I have this kind of functions:

def mean(iterator):
    n = 0
    sum = 0.
    for i in iterator:
      sum += i
      n += 1
    return sum / float(n)

I have also many other similar functions (var, size, ...)

Now I have an iterator iterating throught the data: iter_data. I can compute all the quantities I want: m = mean(iter_data); v = var(iter_data) and so on, but the problem is that I am iterating many times and this is expensive in my case. Actually the I/O is the most expensive part.

So the question is: can I compute my quantities m, v, ... iterating only one time over iter_data keeping separate the functions mean, var, ... so that it is easy to add new ones?

What I need is something similar to boost::accumulators

Answer 1

For example use objects and callbacks like:

class Counter():
    def __init__(self):
        self.n = 0
    def __call__(self, i):
        self.n += 1

class Summer():
    def __init__(self):
        self.sum = 0
    def __call__(self, i):
        self.sum += i


def process(iterator, callbacks):
    for i in iterator:
        for f in callbacks: f(i)

counter = Counter()
summer = Summer()
callbacks = [counter, summer]
iterator = xrange(10) # testdata
process(iterator, callbacks)

# process results from callbacks
n = counter.n
sum = summer.sum

This is easily extendible and iterates the data only once.

Answer 2

You can use itertools.tee and generator magic (I say magic because it's not exactly nice and readable):

import itertools

def mean(iterator):
    n = 0
    sum = 0.
    for i in iterator:
         sum += i
         n += 1
         yield
    yield sum / float(n)

def multi_iterate(funcs, iter_data):
    iterators = itertools.tee(iter_data, len(funcs))
    result_iterators = [func(values) for func, values in zip(funcs, iterators)]
    for results in itertools.izip(*result_iterators):
        pass
    return results

mean_result, var_result = multi_iterate([mean, var], iter([10, 20, 30]))

print(mean_result)    # 20.0

By the way, you can write mean in a simpler way:

def mean(iterator):
    total = 0.
    for n, item in enumerate(iterator, 1):
         total += i
         yield
    yield total / n

You shouldn't name variables sum because that shadows the built-in function with the same name.