Why the staggering overhead [50X] of contextlib and the With statement in Python and what to do about it

Question

In the process of hunting down performance bugs I finally identified that the source of the problem is the contextlib wrapper. The overhead is quite staggering and I did not expect that to be the source of the slowdown. The slowdown is in the range of 50X, I cannot afford to have that in a loop. I sure would have appreciated a warning in the docs if it has the potential of slowing things down so significantly.

It seems this has been known since 2010 https://gist.github.com/bdarnell/736778

It has a set of benchmarks you can try. Please change fn to fn() in simple_catch() before running. Thanks, DSM for pointing this out.

I am surprised that the situation has not improved since those times. What can I do about it? I can drop down to try/except, but I hope there are other ways to deal with it.

Answer 1

Here are some new timings:

import contextlib
import timeit

def work_pass():
    pass

def work_fail():
    1/0

def simple_catch(fn):
    try:
        fn()
    except Exception:
        pass

@contextlib.contextmanager
def catch_context():
    try:
        yield
    except Exception:
        pass

def with_catch(fn):
    with catch_context():
        fn()

class ManualCatchContext(object):
    def __enter__(self):
        pass

    def __exit__(self, exc_type, exc_val, exc_tb):
        return True

def manual_with_catch(fn):
    with ManualCatchContext():
        fn()

preinstantiated_manual_catch_context = ManualCatchContext() 
def manual_with_catch_cache(fn):
    with preinstantiated_manual_catch_context:
        fn()

setup = 'from __main__ import simple_catch, work_pass, work_fail, with_catch, manual_with_catch, manual_with_catch_cache'
commands = [
    'simple_catch(work_pass)',
    'simple_catch(work_fail)',
    'with_catch(work_pass)',
    'with_catch(work_fail)',
    'manual_with_catch(work_pass)',
    'manual_with_catch(work_fail)',
    'manual_with_catch_cache(work_pass)',
    'manual_with_catch_cache(work_fail)',
    ]
for c in commands:
    print c, ': ', timeit.timeit(c, setup)

I've made simple_catch actually call the function and I've added two new benchmarks.

Here's what I got:

>>> python2 bench.py
simple_catch(work_pass) :  0.413918972015
simple_catch(work_fail) :  3.16218209267
with_catch(work_pass) :  6.88726496696
with_catch(work_fail) :  11.8109841347
manual_with_catch(work_pass) :  1.60508012772
manual_with_catch(work_fail) :  4.03651213646
manual_with_catch_cache(work_pass) :  1.32663416862
manual_with_catch_cache(work_fail) :  3.82525682449
python2 p.py.py  33.06s user 0.00s system 99% cpu 33.099 total

And for PyPy:

>>> pypy bench.py
simple_catch(work_pass) :  0.0104489326477
simple_catch(work_fail) :  0.0212869644165
with_catch(work_pass) :  0.362847089767
with_catch(work_fail) :  0.400238037109
manual_with_catch(work_pass) :  0.0223228931427
manual_with_catch(work_fail) :  0.0208241939545
manual_with_catch_cache(work_pass) :  0.0138869285583
manual_with_catch_cache(work_fail) :  0.0213649272919

The overhead is much smaller than you claimed. Further, the only overhead PyPy doesn't seem to be able to remove relative to the try...catch for the manual variant is object creation, which is trivially removed in this case.

---

Unfortunately with is way too involved for good optimization by CPython, especially with regards to contextlib which even PyPy finds hard to optimize. This is normally OK because although object creation + a function call + creating a generator is expensive, it's cheap compared to what is normally done.

If you are sure that with is causing most of your overhead, convert the context managers into cached instances like I have. If that's still too much overhead, you've likely got a bigger problem with how your system is designed. Consider making the scope of the withs bigger (not normally a good idea, but acceptable if need be).

---

Also, PyPy. Dat JIT be fast.