Opening multiple (an unspecified number) of files at once and ensuring they are correctly closed

Question

I am aware that I can open multiple files with something like,

with open('a', 'rb') as a, open('b', 'rb') as b:

But I have a situation where I have a list of files to open and am wondering what the preferred method is of doing the same when the number of files is unknown in advance. Something like,

with [ open(f, 'rb') for f in files ] as fs:

(but this fails with an AttributeError since list doesn't implement __exit__)

I don't mind using something like,

try:
    fs = [ open(f, 'rb') for f in files ]

    ....

finally:
    for f in fs:
        f.close()

But am not sure what will happen if some files throw when trying to open them. Will fs be properly defined, with the files that did manage to open, in the finally block?

Answer 1

No, your code wouldn't initialise fs unless all open() calls completed successfully. This should work though:

fs = []
try:
    for f in files:
        fs.append(open(f, 'rb'))

    ....

finally:
    for f in fs:
        f.close()

Note also that f.close() could fail so you may want to catch and ignore (or otherwise handle) any failures there.

Answer 2

Sure, why not, Here's a recipe that should do it. Create a context manager 'pool' that can enter an arbitrary number of contexts (by calling it's enter() method) and they will be cleaned up at the end of the end of the suite.

class ContextPool(object):
    def __init__(self):
        self._pool = []

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_value, exc_tb):
        for close in reversed(self._pool):
            close(exc_type, exc_value, exc_tb)

    def enter(self, context):
        close = context.__exit__
        result = context.__enter__()
        self._pool.append(close)
        return result

For example:

>>> class StubContextManager(object):
...     def __init__(self, name):
...         self.__name = name
...     def __repr__(self):
...         return "%s(%r)" % (type(self).__name__, self.__name)
... 
...     def __enter__(self):
...          print "called %r.__enter__()" % (self)
... 
...     def __exit__(self, *args):
...          print "called %r.__exit__%r" % (self, args)
... 
>>> with ContextPool() as pool:
...     pool.enter(StubContextManager("foo"))
...     pool.enter(StubContextManager("bar"))
...     1/0
... 
called StubContextManager('foo').__enter__()
called StubContextManager('bar').__enter__()
called StubContextManager('bar').__exit__(<type 'exceptions.ZeroDivisionError'>, ZeroDivisionError('integer division or modulo by zero',), <traceback object at 0x02958648>)
called StubContextManager('foo').__exit__(<type 'exceptions.ZeroDivisionError'>, ZeroDivisionError('integer division or modulo by zero',), <traceback object at 0x02958648>)

Traceback (most recent call last):
  File "<pyshell#67>", line 4, in <module>
    1/0
ZeroDivisionError: integer division or modulo by zero
>>> 

Caveats: context managers aren't supposed to raise exceptions in their __exit__() methods, but if they do, this recipe doesn't do the cleanup for all the context managers. Similarly, even if every context manager indicates that an exception should be ignored (by returning True from their exit methods), this will still allow the exception to be raised.