timeit module - get fastest and slowest loop

Question

I would like to do some benchmarking with Python's timeit module. Is there a way to make timeit return the times for the slowest as well as the fastest loop out of 1000 loops?

Answer 1

timeit returns "best of 3" i.e., there are two parameters: one specifies number of iterations in a loop, another how many times to repeat the loop. The result that is passed to min() is the time per loop, not per iteration of the loop.

The point of repeating the loop is to exclude the influence from other processes on the same system -- from the docs (help('timeit')):

The best thing to do when accurate timing is necessary is to repeat the timing a few times and use the best time. The -r option is good for this; the default of 3 repetitions is probably enough in most cases.

It doesn't make sense to repeat the measarements 1000 times. You probably meant to specify 1000 iteration for a single loop (default is 1000000).

Only the fastest loop (minimum time) is useful -- from help('timeit.Timer.repeat'):

Note: it's tempting to calculate mean and standard deviation from the result vector and report these. However, this is not very useful. In a typical case, the lowest value gives a lower bound for how fast your machine can run the given code snippet; higher values in the result vector are typically not caused by variability in Python's speed, but by other processes interfering with your timing accuracy. So the min() of the result is probably the only number you should be interested in. After that, you should look at the entire vector and apply common sense rather than statistics._{emphasis is mine}

i.e., the slowest loop indicates how much other processes can interfere with the measurements.

#!/usr/bin/env python
import timeit

def your_function():
    "do something"

t = timeit.Timer(your_function)
# repeat 10 times, 1000000 times through the loop
repeat, number = 10, 1000000
r = t.repeat(repeat, number) 
best, worse = min(r), max(r)
print("{number} loops, best of {repeat}: {best:.3g} seconds per loop, "
     "worse of {repeat}: {worse:.3g} seconds per loop".format(**vars()))

Answer 2

Use the repeat function instead of timeit, which return a list of times.