how do I make a 2.7 python context manager threadsafe

Question

I have a large python application which is running on a Django service. I need to turn off permission tests for certain operations so I created this context manager:

class OverrideTests(object):

    def __init__(self):
        self.override = 0

    def __enter__(self):
        self.override += 1

    # noinspection PyUnusedLocal
    def __exit__(self, exc_type, exc_val, exc_tb):
        self.override -= 1
        assert not self.override < 0

    @property
    def overriding(self):
        return self.override > 0

override_tests = OverrideTests()

Various parts of the application can then overide the tests using the context manager:

with override_tests:
    do stuff
    ...

Within the do stuff, the above context manager may be used multiple times in different functions. The use of the counter keeps this under control and it seems to work fine... until threads get involved.

Once there are threads involved, the global context manager gets re-used and as a result, tests may be incorrectly over-ridden.

Here is a simple test case - this works fine if the thread.start_new_thread(do_id, ()) line is replaced with a simple do_it but fails spectacularly as shown:

def stat(k, expected):
    x = '.' if override_tests.overriding == expected else '*'
    sys.stdout.write('{0}{1}'.format(k, x))


def do_it_inner():
    with override_tests:
        stat(2, True)
    stat(3, True)  # outer with context makes this true


def do_it():
    with override_tests:
        stat(1, True)
        do_it_inner()
    stat(4, False)


def do_it_lots(ntimes=10):
    for i in range(ntimes):
        thread.start_new_thread(do_it, ())

How can I make this context manager thread safe so that in each Python thread, it is consistently used even though it is re-entrant?

Answer 1

Here is a way that seems to work: make your OverrideTests class a subclass of threading.local. For safety, you should then call the superclass __init__ in your __init__ (although it seems to work even if you don't):

class OverrideTests(threading.local):

    def __init__(self):
        super(OverrideTests, self).__init__()
        self.override = 0

    # rest of class same as before

override_tests = OverrideTests()

Then:

>>> do_it_lots()
1.1.1.2.2.1.1.1.1.1.1.3.3.2.2.2.2.2.2.4.4.3.1.3.3.3.3.4.3.2.4.4.2.4.3.4.4.4.3.4.

However, I wouldn't put money on this not failing in some kind of corner case, especially if your real application is more complex than the example you showed here. Ultimately, you really should rethink your design. In your question, you are focusing on how to "make the context-manager threadsafe". But the real problem is not just with your context manager but with your function (stat in your example). stat is relying on global state (the global override_tests), which is inherently fragile in a threaded environment.