Will a Python generator be garbage collected if it will not be used any more but hasn't reached StopIteration yet?

Question

When a generator is not used any more, it should be garbage collected, right? I tried the following code but I am not sure which part I was wrong.

import weakref import gc  def countdown(n):     while n:         yield n         n-=1  cd = countdown(10) cdw = weakref.ref(cd)() print cd.next() gc.collect() print cd.next() gc.collect() print cdw.next() 

On the second last line, I called garbage collector and since there is no call to cd any more. gc should free cd right. But when I call cdw.next(), it is still printing 8. I tried a few more cdw.next(), it could successfully print all the rest until StopIteration.

I tried this because I wanted to understand how generator and coroutine work. On slide 28 of David Beazley's PyCon presentation "A Curious Course on Coroutines and Concurrency", he said that a coroutine might run indefinitely and we should use .close() to shut it down. Then he said that garbage collector will call .close(). In my understanding, once we called .close() ourselves, gc will call .close() again. Will gc receive a warning that it can't call .close() on an already closed coroutine?

Thanks for any inputs.

Answer 1

Due to the dynamic nature of python, the reference to cd isn't freed until you reach the end of the current routine because (at least) the Cpython implementation of python doesn't "read ahead". (If you don't know what python implementation you're using, it's almost certainly "Cpython"). There are a number of subtleties that would make that virtually impossible for the interpreter to determine whether an object should be free if it still exists in the current namespace in the general case (e.g. you can still reach it by a call to locals()).

In some less general cases, other python implementations may be able to free an object before the end of the current stack frame, but Cpython doesn't bother.

Try this code instead which demonstrates that the generator is free to be cleaned up in Cpython:

import weakref def countdown(n):     while n:         yield n         n-=1  def func():     a = countdown(10)     b = weakref.ref(a)     print next(a)     print next(a)     return b  c = func() print c() 

Objects (including generators) are garbage collected when their reference count reaches 0 (in Cpython -- Other implementations may work differently). In Cpython, reference counts are only decremented when you see a del statement, or when an object goes out of scope because the current namespace changes.

The important thing is that once there are no more references to an object, it is free to be cleaned up by the garbage collector. The details of how the implementation determines that there are no more references are left to the implementers of the particular python distribution you're using.