Weird callback execution order in Twisted?

Question

Consider the following code:

from twisted.internet.defer import Deferred

d1 = Deferred()
d2 = Deferred()

def f1(result):
    print 'f1',

def f2(result):
    print 'f2',

def f3(result):
    print 'f3',

def fd(result):
    return d2

d1.addCallback(f1)
d1.addCallback(fd)
d1.addCallback(f3)

#/BLOCK====
d2.addCallback(f2)
d1.callback(None)
#=======BLOCK/

d2.callback(None)

This outputs what I would expect:

f1 f2 f3

However when I swap the order of the statements in BLOCK to

#/BLOCK====
d1.callback(None)
d2.addCallback(f2)
#=======BLOCK/

i.e. Fire d1 before adding the callback to d2, I get:

f1 f3 f2

I don't see why the time of firing of the deferreds should influence the callback execution order.
Is this an issue with Twisted or does this make sense in some way?

Answer 1

tl;dr — When you return a deferred (d2) from a callback (fd), it's inserted into the callback chain of whatever deferred (d1) called fd. This is done by adding a continuation of d1's callback chain as a callback on d2, so if you add a callback to d2 after firing d1, it gets tacked on after d1's continuation.

---

I think you're missing the fact that deferreds aren't asynchronous in-and-of themselves, they're just a more structured way to chain callbacks and error handlers in asynchronous code. The callbacks aren't called at some unspecified later time, they're called when the deferred is fired. If you want this to be later, you'll need to integrate your calls with a reactor.

So maybe that makes the answer to this a little more obvious:

I don't see why the time of firing of the deferreds should influence the callback execution order.

It will if you change the callbacks after you fire the deferred.

Explanation of your code

In the first example:

1. d1.callback(None) causes d1 to be fired
2. f1 is called (prints "f1")
3. fd is called (returns d2)

Now things stop, because d2 has been inserted in the callback chain for d1 between f1 and f3, but has not yet been fired. Then...

1. d2.callback(None) causes d2 to be fired
2. f2 is called (prints "f2")
3. d1's callback chain resumes; so f3 is called (prints "f3")

In the second example

1. d1.callback(None) causes d1 to be fired
2. f1 is called (prints "f1")
3. fd is called (returns d2)

Here d2 is again inserted into the callback chain. The way this is done is by adding a continuation of the current callback chain as a callback to d2. So even though you explicitly add f2 as a callback on d2, it's added after the continuation of d1's callback chain. Hence...

1. d2.callback(None) causes d2 to be fired; this causes d1's callback chain to be continued, starting with...
2. f3 is called (prints "f3")
3. d2's next callback in it's chain is f2, so...
4. f2 is called (prints "f2")