Yielding from within with statement and __exit__ method of context manager

Question

Consider following Python 2.x code snippet.

from __future__ import print_function


class myfile(file):
    def __exit__(self, *excinfo):
        print("__exit__ called")
        super(myfile, self).__exit__(*excinfo)


def my_generator(file_name):
    with myfile(file_name) as fh:
        for line in fh:
            yield line.strip()


gen = my_generator('file.txt')
print(next(gen))
print("Before del")
del gen
print("After del")

Output of this script (given file.txt has more than one line) is:

Line 1 from file
Before del
__exit__ called
After del

I'm interested about __exit__ call specifically.

What triggers execution of his method? For what we know, code never left with statement (it "stopped" after yield statement and never continued). Is it guaranteed that __exit__ will be called when reference count of generator drops to 0?

Answer 1

On reclamation of a generator object, Python calls its close method, raising a GeneratorExit exception at the point of its last yield if it wasn't already finished executing. As this GeneratorExit propagates, it triggers the __exit__ method of the context manager you used.

This was introduced in Python 2.5, in the same PEP as send and yield expressions. Before then, you couldn't yield inside a try with a finally, and if with statements had existed pre-2.5, you wouldn't have been able to yield inside one either.

Answer 2

To add to @user2357112's answer, the with block breaks when an exception is raised inside of it. This exception is passed to the __exit__ method of the object that was created for the context.

The file class seems to pass silently the GeneratorExit exception, since nothing signals it. However, if you print argc in your myfile.__exit__ method, you will see that the context was not closed naturally:

class myfile(file):
    def __exit__(self, *excinfo):
        print("__exit__ called")
        print(excinfo[0]) # Print the reason why the context exited
        super(myfile, self).__exit__(*excinfo)

Output of your script:

Line 1 from file
Before del
__exit__ called
<type 'exceptions.GeneratorExit'>
After del