Lifespan of open handles in a python generator body

Question

I have a question about generators and/or python's execution model.

Given something like:

def g(filename):
    with open(filename) as handle:
        for line in handle:
            yield some_trasformation_on(line)

My main confusion is: What happens to handle if

res = g()
print(next(res))

is called only once? Does the handle stay open for the life of the program? What if we have scarce resources being allocated in a generator body? Threads? Database handles?

Perhaps I am considering generators to be like functions under the hood.

Answer 1

Answer with the code as written

As described, the file will stay open for the life of the program. While you have a reference to the res variable, it keeps the generator stackframe is alive, so the only way to get the with-statement to close the file is to keep calling next() until the for-loop ends normally.

How to clean-up the generator

The tool designed for generator clean-up is generator.close() which raises a GeneratorExit inside the generator.

Here is some working code to show how it could be done:

def genfunc(filename):
    try:
        with open(filename) as datafile:
            for line in datafile:
                yield len(line)
    finally:
        print('Status: %s' % datafile.closed)

And here is a sample session showing the closing operation actually takes place:

>>> g = genfunc('somefile.txt')
>>> next(g)
23
>>> g.close()
Status: True