Exception handling in Python Tornado

Question

I am trying to handle exception occurred in AsyncClient.fetch in this way:

from tornado.httpclient import AsyncHTTPClient
from tornado.httpclient import HTTPRequest
from tornado.stack_context import ExceptionStackContext
from tornado import ioloop

def handle_exc(*args):
    print('Exception occured')
    return True

def handle_request(response):
    print('Handle request')

http_client = AsyncHTTPClient()

with ExceptionStackContext(handle_exc):
    http_client.fetch('http://some123site.com', handle_request)

ioloop.IOLoop.instance().start()

and see next output:

WARNING:root:uncaught exception
Traceback (most recent call last):
  File "/home/crchemist/python-3.2/lib/python3.2/site-packages/tornado-2.0-py3.2.egg/tornado/simple_httpclient.py", line 259, in cleanup
    yield
  File "/home/crchemist/python-3.2/lib/python3.2/site-packages/tornado-2.0-py3.2.egg/tornado/simple_httpclient.py", line 162, in __init__
    0, 0)
socket.gaierror: [Errno -5] No address associated with hostname
Handle request

What am I doing wrong?

Answer 1

According to the Tornado documentation:

If an error occurs during the fetch, the HTTPResponse given to the callback has a non-None error attribute that contains the exception encountered during the request. You can call response.rethrow() to throw the exception (if any) in the callback.

from tornado.httpclient import AsyncHTTPClient
from tornado.httpclient import HTTPRequest
from tornado.stack_context import ExceptionStackContext
from tornado import ioloop

import traceback

def handle_exc(*args):
    print('Exception occured')
    return True

def handle_request(response):
    if response.error is not None:
        with ExceptionStackContext(handle_exc):
            response.rethrow()
    else:
        print('Handle request')

http_client = AsyncHTTPClient()

http_client.fetch('http://some123site.com', handle_request)
http_client.fetch('http://google.com', handle_request)

ioloop.IOLoop.instance().start()

---

The message you're seeing on the console is only a warning (sent through logging.warning). It's harmless, but if it really bothers you, see the logging module for how to filter it.

Answer 2

I don't know Tornado at all, but I gave a look and you simply can't catch exceptions that way. The exception is generated in the constructor of _HTTPConnection(), and most of the code in that constructor is already wrapped by a different stack context:

    with stack_context.StackContext(self.cleanup):
        parsed = urlparse.urlsplit(_unicode(self.request.url))
        [...]

So basically whenever an exception is generated there (gaierror in your example), it is caught already and handled through self.cleanup, that in turns generate a 599 response AFAICT:

@contextlib.contextmanager
def cleanup(self):
    try:
        yield
    except Exception, e:
        logging.warning("uncaught exception", exc_info=True)
        self._run_callback(HTTPResponse(self.request, 599, error=e,
                            request_time=time.time() - self.start_time,
                            ))

Not sure if this answers your question.