No stack trace seen with 'Unhandled error in Deferred' when using Python Twisted

Question

Having read through http://krondo.com/an-introduction-to-asynchronous-programming-and-twisted/, I have a basic understanding of Twisted. I have built a testing infrastructure that a few other people also use. From time to time, we experience 'Unhandled error in Deferred' type errors if there are bugs or typos in the code. The problem is that there isn't always an accompanying stack trace with these errors making it hard for people to debug.

I have reproduced the problem with the following simple code:

from twisted.internet import defer, reactor
from twisted.internet.task import deferLater


def sleep(delay):
    """Twisted safe sleep.

    When using Twisted, it is not safe to call time.sleep().  So
    we have this function to emulate the behavior.

    """
    return deferLater(reactor, delay, lambda: None)


@defer.inlineCallbacks
def run_test():
    print 'run_test: start'
    bug()
    yield sleep(1)
    print 'run_test: stop'


@defer.inlineCallbacks
def run_tests():

    def err(arg):
        print 'err', arg
        return arg

    def success(arg):
        print 'success', arg
        return arg

    d = run_test()
    #d.addCallbacks(success, err)

    try:
        yield d
    finally:
        reactor.stop()


reactor.callWhenRunning(run_tests)
reactor.run()

When I run this code, I see the following output:

run_test: start
Unhandled error in Deferred:

And if I uncomment the addCallbacks() line above, then I get some stack trace information:

run_test: start
err [Failure instance: Traceback: <type 'exceptions.NameError'>: global name 'bug' is not defined
/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py:1406:unwindGenerator
/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py:1260:_inlineCallbacks
tmp.py:34:run_tests
/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py:1406:unwindGenerator
--- <exception caught here> ---
/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py:1260:_inlineCallbacks
tmp.py:18:run_test
]
Unhandled error in Deferred:

My question is, is there some way to get the stack trace without having to add callbacks at all defer sites?

Answer 1

The Deferred class has tons of try/except magic, to the point that the errors are pretty well abstracted from the end user and only easily attained via addErrback. If errors are occurring, it's a sign you have broken functionality. In synchronous apps, you might encapsulate the "problem section" in a try/except block. In the case of inlineCallbacks, the same technique can be utilized:

try:
    run_test()
except Exception as err:
    # handle error here
finally:
    reactor.stop()

Since the bug occurs in the run_test() function, catch the exception when running that function, then handle the error according to your requirements.

However, if you don't plan to "handle" errors, but rather you want a record that an error has occurred, then you should consider using Twisted logger functionality. This will catch your unhandled tracebacks and log them somewhere.