How can Twisted Deferred errors without errbacks be tested with trial?

Question

I have some Twisted code which creates multiple chains of Deferreds. Some of these may fail without having an errback which puts them back on the callback chain. I haven't been able to write a unit test for this code - the failing Deferred causes the test to fail after the test code has completed. How can I write a passing unit test for this code? Is it expected that every Deferred which could fail in normal operation should have an errback at the end of the chain which puts it back on the callback chain?

The same thing happens when there's a failed Deferred in a DeferredList, unless I create the DeferredList with consumeErrors. This is the case even when the DeferredList is created with fireOnOneErrback and is given an errback which puts it back on the callback chain. Are there any implications for consumeErrors besides suppressing test failures and error logging? Should every Deferred which may fail without an errback be put a DeferredList?

Example tests of example code:

from twisted.trial import unittest
from twisted.internet import defer

def get_dl(**kwargs):
    "Return a DeferredList with a failure and any kwargs given."
    return defer.DeferredList(
        [defer.succeed(True), defer.fail(ValueError()), defer.succeed(True)],
        **kwargs)

def two_deferreds():
    "Create a failing Deferred, and create and return a succeeding Deferred."
    d = defer.fail(ValueError())
    return defer.succeed(True)


class DeferredChainTest(unittest.TestCase):

    def check_success(self, result):
        "If we're called, we're on the callback chain."        
        self.fail()

    def check_error(self, failure):
        """
        If we're called, we're on the errback chain.
        Return to put us back on the callback chain.
        """
        return True

    def check_error_fail(self, failure):
        """
        If we're called, we're on the errback chain.
        """
        self.fail()        

    # This fails after all callbacks and errbacks have been run, with the
    # ValueError from the failed defer, even though we're
    # not on the errback chain.
    def test_plain(self):
        """
        Test that a DeferredList without arguments is on the callback chain.
        """
        # check_error_fail asserts that we are on the callback chain.
        return get_dl().addErrback(self.check_error_fail)

    # This fails after all callbacks and errbacks have been run, with the
    # ValueError from the failed defer, even though we're
    # not on the errback chain.
    def test_fire(self):
        """
        Test that a DeferredList with fireOnOneErrback errbacks on failure,
        and that an errback puts it back on the callback chain.
        """
        # check_success asserts that we don't callback.
        # check_error_fail asserts that we are on the callback chain.
        return get_dl(fireOnOneErrback=True).addCallbacks(
            self.check_success, self.check_error).addErrback(
            self.check_error_fail)

    # This succeeds.
    def test_consume(self):
        """
        Test that a DeferredList with consumeErrors errbacks on failure,
        and that an errback puts it back on the callback chain.
        """
        # check_error_fail asserts that we are on the callback chain.
        return get_dl(consumeErrors=True).addErrback(self.check_error_fail)

    # This succeeds.
    def test_fire_consume(self):
        """
        Test that a DeferredList with fireOnOneCallback and consumeErrors
        errbacks on failure, and that an errback puts it back on the
        callback chain.
        """
        # check_success asserts that we don't callback.
        # check_error_fail asserts that we are on the callback chain.
        return get_dl(fireOnOneErrback=True, consumeErrors=True).addCallbacks(
            self.check_success, self.check_error).addErrback(
            self.check_error_fail)

    # This fails after all callbacks and errbacks have been run, with the
    # ValueError from the failed defer, even though we're
    # not on the errback chain.
    def test_two_deferreds(self):
        # check_error_fail asserts that we are on the callback chain.        
        return two_deferreds().addErrback(self.check_error_fail)

Answer 1

There are two important things about trial related to this question.

First, a test method will not pass if a Failure is logged while it is running. Deferreds which are garbage collected with a Failure result cause the Failure to be logged.

Second, a test method which returns a Deferred will not pass if the Deferred fires with a Failure.

This means that neither of these tests can pass:

def test_logit(self):
    defer.fail(Exception("oh no"))

def test_returnit(self):
    return defer.fail(Exception("oh no"))

This is important because the first case, the case of a Deferred being garbage collected with a Failure result, means that an error happened that no one handled. It's sort of similar to the way Python will report a stack trace if an exception reaches the top level of your program.

Likewise, the second case is a safety net provided by trial. If a synchronous test method raises an exception, the test doesn't pass. So if a trial test method returns a Deferred, the Deferred must have a success result for the test to pass.

There are tools for dealing with each of these cases though. After all, if you couldn't have a passing test for an API that returned a Deferred that fired with a Failure sometimes, then you could never test your error code. That would be a pretty sad situation. :)

So, the more useful of the two tools for dealing with this is TestCase.assertFailure. This is a helper for tests that want to return a Deferred that's going to fire with a Failure:

def test_returnit(self):
    d = defer.fail(ValueError("6 is a bad value"))
    return self.assertFailure(d, ValueError)

This test will pass because d does fire with a Failure wrapping a ValueError. If d had fired with a success result or with a Failure wrapping some other exception type, then the test would still fail.

Next, there's TestCase.flushLoggedErrors. This is for when you're testing an API that's supposed to log an error. After all, sometimes you do want to inform an administrator that there's a problem.

def test_logit(self):
    defer.fail(ValueError("6 is a bad value"))
    gc.collect()
    self.assertEquals(self.flushLoggedErrors(ValueError), 1)

This lets you inspect the failures which got logged to make sure your logging code is working properly. It also tells trial not to worry about the things you flushed, so they'll no longer cause the test to fail. (The gc.collect() call is there because the error isn't logged until the Deferred is garbage collected. On CPython, it'll be garbage collected right away due to the reference counting GC behavior. However, on Jython or PyPy or any other Python runtime without reference counting, you can't rely on that.)

Also, since garbage collection can happen at pretty much any time, you might sometimes find that one of your tests fails because an error is logged by a Deferred created by an earlier test being garbage collected during the execution of the later test. This pretty much always means your error handling code is incomplete in some way - you're missing an errback, or you failed to chain two Deferreds together somewhere, or you're letting your test method finish before the task it started actually finishes - but the way the error is reported sometimes makes it hard to track down the offending code. Trial's --force-gc option can help with this. It causes trial to invoke the garbage collector between each test method. This will slow down your tests significantly, but it should cause the error to be logged against the test which is actually triggering it, not an arbitrary later test.