Why contextmanager is slow

Question

A coworker pointed out to me, that with statement can be slow. So I measured and indeed it takes 20 times longer to get value from a contextmanager function than from a generator in Python 2.7 and even 200 times longer in PyPy 2.6.

Why is it so? Is it possible to rewrite contextlib.contextmanager() to run faster?

For the reference:

def value_from_generator():
    def inner(): yield 1

    value, = inner()
    return value

def value_from_with():
    @contextmanager
    def inner(): yield 1

    with inner() as value:
        return value

And timings:

$ python -m timeit 'value_from_generator()'
10000000 loops, best of 3: 0.169 usec per loop

$ python -m timeit 'value_from_with()'
100000 loops, best of 3: 3.04 usec per loop

Answer 1

Using profiler and source of contextlib, i found:

value_from_with:
 ncalls  tottime  cumtime  filename:lineno(function)
1000000    1.415    4.802  value_from_with  # 1sec more than value_from_generator, likely caused by with statement
1000000    1.115    1.258  contextlib.py:37(__init__)  # better doc string of context manager instance
1000000    0.656    0.976  contextlib.py:63(__exit__)  # optional exception handling
1000000    0.575    1.833  contextlib.py:124(helper)  # "wrapped" in decorator
2000000    0.402    0.604  {built-in method next}  # why it's so expensive?
1000000    0.293    0.578  contextlib.py:57(__enter__)  # a next() call to the generator in try&except block (just for error msg)
2000000    0.203    0.203  inner1
1000000    0.143    0.143  {built-in method getattr}  # better doc string, called by __init__

value_from_generator:
 ncalls  tottime  cumtime  filename:lineno(function)
1000000    0.416    0.546  value_from_generator
2000000    0.130    0.130  inner2

It told us: unpacking from generator is faster than using next(); function call is expensive; exception handling is expensive...so the comparison is unfair, and this profiling is just for fun.

It also told us that every time "with" block is executed a instance of context manager is created (almost unavoidable). Besides this, contextmanager did some job to convinient us. If you really want to optimize it, you can write a context manager class instead of using the decorator

profiled code:

def inner1(): yield 1

def value_from_generator():
    value, = inner1()
    return value

# inner should not be created again and again
@contextmanager
def inner2(): yield 1

def value_from_with():
    with inner2() as value:
        return value