Pythonic way to compose context managers for objects owned by a class

Question

It's typical to require for some task multiple objects which have resources to be explicitly released - say, two files; this is easily done when the task is local to a function using nested with blocks, or - even better - a single with block with multiple with_item clauses:

with open('in.txt', 'r') as i, open('out.txt', 'w') as o:
    # do stuff

OTOH, I still struggle to understand how this is supposed to work when such objects aren't just local to a function scope, but owned by a class instance - in other words, how context managers compose.

Ideally I'd like to do something like:

class Foo:
    def __init__(self, in_file_name, out_file_name):
        self.i = WITH(open(in_file_name, 'r'))
        self.o = WITH(open(out_file_name, 'w'))

and have Foo itself turn into a context manager that handles i and o, such that when I do

with Foo('in.txt', 'out.txt') as f:
    # do stuff

self.i and self.o are taken care of automatically as you would expect.

I tinkered about writing stuff such as:

class Foo:
    def __init__(self, in_file_name, out_file_name):
        self.i = open(in_file_name, 'r').__enter__()
        self.o = open(out_file_name, 'w').__enter__()

    def __enter__(self):
        return self

    def __exit__(self, *exc):
        self.i.__exit__(*exc)
        self.o.__exit__(*exc)

but it's both verbose and unsafe against exceptions occurring in the constructor. After searching for a while, I found this 2015 blog post, which uses contextlib.ExitStack to obtain something very similar to what I'm after:

class Foo(contextlib.ExitStack):
    def __init__(self, in_file_name, out_file_name):
        super().__init__()
        self.in_file_name = in_file_name
        self.out_file_name = out_file_name

    def __enter__(self):
        super().__enter__()
        self.i = self.enter_context(open(self.in_file_name, 'r')
        self.o = self.enter_context(open(self.out_file_name, 'w')
        return self

This is pretty satisfying, but I'm perplexed by the fact that:

• I find nothing about this usage in the documentation, so it doesn't seem to be the "official" way to tackle this problem;
• in general, I find it extremely difficult to find information about this issue, which makes me think I'm trying to apply an unpythonic solution to the problem.

---

Some extra context: I work mostly in C++, where there is no distinction between the block-scope case and the object-scope case for this issue, as this kind of cleanup is implemented inside the destructor (think __del__, but invoked deterministically), and the destructor (even if not explicitly defined) automatically invokes the destructors of the subobjects. So both:

{
    std::ifstream i("in.txt");
    std::ofstream o("out.txt");
    // do stuff
}

and

struct Foo {
    std::ifstream i;
    std::ofstream o;

    Foo(const char *in_file_name, const char *out_file_name) 
        : i(in_file_name), o(out_file_name) {}
}

{
    Foo f("in.txt", "out.txt");
}

do all the cleanup automatically as you generally want.

I'm looking for a similar behavior in Python, but again, I'm afraid I'm just trying to apply a pattern coming from C++, and that the underlying problem has a radically different solution that I can't think of.

---

So, to sum it up: what is the Pythonic solution to the problem of having an object who owns objects that require cleanup become a context-manager itself, calling correctly the __enter__/__exit__ of its children?

Answer 1

I think contextlib.ExitStack is Pythonic and canonical and it's the appropriate solution to this problem. The rest of this answer tries to show the links I used to come to this conclusion and my thought process:

Original Python enhancement request

https://bugs.python.org/issue13585

The original idea + implementation was proposed as a Python standard library enhancement with both reasoning and sample code. It was discussed in detail by such core developers as Raymond Hettinger and Eric Snow. The discussion on this issue clearly shows the growth of the original idea into something that is applicable for the standard library and is Pythonic. Attempted summarization of the thread is:

nikratio originally proposed:

I'd like to propose addding the CleanupManager class described in http://article.gmane.org/gmane.comp.python.ideas/12447 to the contextlib module. The idea is to add a general-purpose context manager to manage (python or non-python) resources that don't come with their own context manager

Which was met with concerns from rhettinger:

So far, there has been zero demand for this and I've not seen code like it being used in the wild. AFAICT, it is not demonstrably better than a straight-forward try/finally.

As a response to this there was a long discussion about whether there was a need for this, leading to posts like these from ncoghlan:

TestCase.setUp() and TestCase.tearDown() were amongst the precursors to__enter__() and exit(). addCleanUp() fills exactly the same role here - and I've seen plenty of positive feedback directed towards Michael for that addition to the unittest API... ...Custom context managers are typically a bad idea in these circumstances, because they make readability worse (relying on people to understand what the context manager does). A standard library based solution, on the other hand, offers the best of both worlds: - code becomes easier to write correctly and to audit for correctness (for all the reasons with statements were added in the first place) - the idiom will eventually become familiar to all Python users... ...I can take this up on python-dev if you want, but I hope to persuade you that the desire is there...

And then again from ncoghlan a little later:

My earlier descriptions here aren't really adequate - as soon as I started putting contextlib2 together, this CleanupManager idea quickly morphed into ContextStack [1], which is a far more powerful tool for manipulating context managers in a way that doesn't necessarily correspond with lexical scoping in the source code.

Examples / recipes / blog posts of ExitStack There are several examples and recipes within the standard library source code itself, which you can see in the merge revision that added this feature: https://hg.python.org/cpython/rev/8ef66c73b1e1

There is also a blog post from the original issue creator (Nikolaus Rath / nikratio) that describes in a compelling way why ContextStack is a good pattern and also provides some usage examples: https://www.rath.org/on-the-beauty-of-pythons-exitstack.html

Answer 2

Your second example is the most straight forward way to do it in Python (i.e., most Pythonic). However, your example still has a bug. If an exception is raised during the second open(),

self.i = self.enter_context(open(self.in_file_name, 'r')
self.o = self.enter_context(open(self.out_file_name, 'w') # <<< HERE

then self.i will not be released when you expect because Foo.__exit__() will not be called unless Foo.__enter__() successfully returns. To fix this, wrap each context call in a try-except that will call Foo.__exit__() when an exception occurs.

import contextlib
import sys

class Foo(contextlib.ExitStack):

    def __init__(self, in_file_name, out_file_name):
        super().__init__()
        self.in_file_name = in_file_name
        self.out_file_name = out_file_name

    def __enter__(self):
        super().__enter__()

        try:
            # Initialize sub-context objects that could raise exceptions here.
            self.i = self.enter_context(open(self.in_file_name, 'r'))
            self.o = self.enter_context(open(self.out_file_name, 'w'))

        except:
            if not self.__exit__(*sys.exc_info()):
                raise

        return self