Memory leak in tornado generator engine with try/finally block when connections are closed

Question

This awesome code, shows memory leak in tornado's gen module, when connections are closed without reading the response:

import gc
from tornado import web, ioloop, gen

class MainHandler(web.RequestHandler):
    @web.asynchronous
    @gen.engine
    def get(self):
        gc.collect()
        print len(gc.garbage)  # print zombie objects count
        self.a = '*' * 500000000  # ~500MB data
        CHUNK_COUNT = 100
        try:
            for i in xrange(CHUNK_COUNT):
                self.write('*' * 10000)  # write ~10KB of data
                yield gen.Task(self.flush)  # wait for reciever to recieve
            print 'finished'
        finally:
            print 'finally'

application = web.Application([
    (r"/", MainHandler),
    ])

application.listen(8888)
ioloop.IOLoop.instance().start()

and now, run a simple test client, multiple times

#!/usr/bin/python
import urllib
urlopen('http://127.0.0.1:8888/')  # exit without reading response

Now, server output shows, incremental memory usage:

0
WARNING:root:Write error on 8: [Errno 104] Connection reset by peer
1
WARNING:root:Read error on 8: [Errno 104] Connection reset by peer
WARNING:root:error on read
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/iostream.py", line 361, in _handle_read
    if self._read_to_buffer() == 0:
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/iostream.py", line 428, in _read_to_buffer
    chunk = self._read_from_socket()
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/iostream.py", line 409, in _read_from_socket
    chunk = self.socket.recv(self.read_chunk_size)
error: [Errno 104] Connection reset by peer
2
ERROR:root:Uncaught exception GET / (127.0.0.1)
HTTPRequest(protocol='http', host='127.0.0.1:8888', method='GET', uri='/', version='HTTP/1.0', remote_ip='127.0.0.1', body='', headers={'Host': '127.0.0.1:8888', 'User-Agent': 'Python-urllib/1.17'})
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/web.py", line 1021, in _stack_context_handle_exception
    raise_exc_info((type, value, traceback))
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/web.py", line 1139, in wrapper
    return method(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/gen.py", line 120, in wrapper
    runner.run()
  File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/gen.py", line 345, in run
    yielded = self.gen.send(next)
  File "test.py", line 10, in get
    self.a = '*' * 500000000
MemoryError
ERROR:root:500 GET / (127.0.0.1) 3.91ms

If you set CHUNK_COUNT to 1, the 10KB of data can be written to OS connection buffer, and 'finished' and 'finally' texts will be printed to console, and because generator is completed, no memory leak occurs.

But the strange part is that if your remove the try/finally block, the problem disappears!! (even with CHUNK_COUNT set to 100)

Is this a bug on CPython or tornado or ...?!

Answer 1

This bug tested with Tornado 2.4.1 (the latest version when this question asked), and reported on https://github.com/facebook/tornado/issues/660 .

The problem fixed in commit https://github.com/facebook/tornado/commit/769bc52e11656788782a6e7a922ef646503f9ab0 and included in Tornado 3.0.