Is it safe to yield from within a "with" block in Python (and why)?

Question

The combination of coroutines and resource acquisition seems like it could have some unintended (or unintuitive) consequences.

The basic question is whether or not something like this works:

def coroutine():     with open(path, 'r') as fh:         for line in fh:             yield line 

Which it does. (You can test it!)

The deeper concern is that with is supposed to be something an alternative to finally, where you ensure that a resource is released at the end of the block. Coroutines can suspend and resume execution from within the with block, so how is the conflict resolved?

For example, if you open a file with read/write both inside and outside a coroutine while the coroutine hasn't yet returned:

def coroutine():     with open('test.txt', 'rw+') as fh:         for line in fh:             yield line  a = coroutine() assert a.next() # Open the filehandle inside the coroutine first. with open('test.txt', 'rw+') as fh: # Then open it outside.     for line in fh:         print 'Outside coroutine: %r' % repr(line) assert a.next() # Can we still use it? 

Update

I was going for write-locked file handle contention in the previous example, but since most OSes allocate filehandles per-process there will be no contention there. (Kudos to @Miles for pointing out the example didn't make too much sense.) Here's my revised example, which shows a real deadlock condition:

import threading  lock = threading.Lock()  def coroutine():     with lock:         yield 'spam'         yield 'eggs'  generator = coroutine() assert generator.next() with lock: # Deadlock!     print 'Outside the coroutine got the lock' assert generator.next() 

Answer 1

I don't really understand what conflict you're asking about, nor the problem with the example: it's fine to have two coexisting, independent handles to the same file.

One thing I didn't know that I learned in response to your question it that there is a new close() method on generators:

close() raises a new GeneratorExit exception inside the generator to terminate the iteration. On receiving this exception, the generator’s code must either raise GeneratorExit or StopIteration.

close() is called when a generator is garbage-collected, so this means the generator’s code gets one last chance to run before the generator is destroyed. This last chance means that try...finally statements in generators can now be guaranteed to work; the finally clause will now always get a chance to run. This seems like a minor bit of language trivia, but using generators and try...finally is actually necessary in order to implement the with statement described by PEP 343.

http://docs.python.org/whatsnew/2.5.html#pep-342-new-generator-features

So that handles the situation where a with statement is used in a generator, but it yields in the middle but never returns—the context manager's __exit__ method will be called when the generator is garbage-collected.

---

Edit:

With regards to the file handle issue: I sometimes forget that there exist platforms that aren't POSIX-like. :)

As far as locks go, I think Rafał Dowgird hits the head on the nail when he says "You just have to be aware that the generator is just like any other object that holds resources." I don't think the with statement is really that relevant here, since this function suffers from the same deadlock issues:

def coroutine():     lock.acquire()     yield 'spam'     yield 'eggs'     lock.release()  generator = coroutine() generator.next() lock.acquire() # whoops!