Tornado graceful shutdown of IOLoop

Question

I'm using the following code to close down my tornado application gracefully (taken from https://gist.github.com/wonderbeyond/d38cd85243befe863cdde54b84505784):

def sig_handler(servers, sig, frame):
    io_loop = tornado.ioloop.IOLoop.instance()

    def stop_loop(deadline):
        now = time.time()
        if now < deadline and (io_loop._callbacks or io_loop._timeouts):
            logging.info('Waiting for next tick')
            print("CALL BACKS")
            print(io_loop._callbacks)
            print("TIMEOUTS")
            print(io_loop._timeouts)
            io_loop.add_timeout(now + 1, stop_loop, deadline)
        else:
            io_loop.stop()
            logging.info("Shutting down.")

    def shutdown():
        logging.info("Stopping http servers")

        # servers is a list of servers to stop
        for s in servers:
           s.stop()

        logging.info("Will shutdown in %s seconds ...",
                     MAX_WAIT_SEC_BEFORE_SHUTDOWN)
        stop_loop(time.time() + MAX_WAIT_SEC_BEFORE_SHUTDOWN)

    logging.warning("Caught signal: %s", sig)
    io_loop.add_callback_from_signal(shutdown)

I set MAX_WAIT_SEC_BEFORE_SHUTDOWN to 10 seconds. Even after closing down the http servers it takes the full 10 seconds everytime to close down the server. I have noted that there are always items in the io_loop._timeouts list E.g:

[<tornado.ioloop._Timeout object at 0x106b90408>, <tornado.ioloop._Timeout object at 0x106b904c8>, ...]

What are the items in io_loop._timeouts? Should I expect this to be an empty list or am I not stopping something that i should have?

Is this shutdown routine normal? Can anyone suggest other code?

Answer 1

Stopping a Tornado HTTPServer does not close all existing connections, it only stops the server from accepting new ones. There is an undocumented close_all_connections method, but it does not distinguish between connections that are idle and those that are currently processing requests, so it is not suitable for use in a graceful shutdown. There is not currently any way to close all connections as they become idle.

Each idle connection maintains a timeout on the IOLoop (the connection will be closed if it is idle for too long, although the default for this is one hour. Pass idle_connection_timeout= to the HTTPServer constructor to change this). In addition, other features of Tornado or other libraries may also create timeouts (in Tornado itself, curl_httpclient and autoreload both create timeouts that are always running), so you cannot in general assume that the number of pending timeouts will ever reach zero.

In general, I recommend just waiting unconditionally for MAX_WAIT_SEC_BEFORE_SHUTDOWN instead of trying to determine whether it is safe to shut down sooner. If you do want to determine when all pending operations have finished, then it is best to keep a count of pending operations yourself (for whatever definition of "operation" you find appropriate) instead of trying to infer this from implementation details of the IOLoop.