deferToThread vs Deferred()

Question

I would like to understand what happens with the below 2 code snippets -

SNIPPET#1

from twisted.internet import threads, defer, reactor

def proc1(a):
    while True:
        print "Proc----------1"

def proc2(a):
    while True:
        print "Proc----------2"

def proc3(a):
    while True:
        print "Proc----------3"

d1 = threads.deferToThread(proc1)
d2 = threads.deferToThread(proc2)
d3 = threads.deferToThread(proc3)

reactor.run()

My understanding is all the threads run parallely and the output is => stdout of all the procs mixed

SNIPPER#2

from twisted.internet import threads, defer, reactor

def proc1(a):
    while True:
        print "Proc----------1"

def proc2(a):
    while True:
        print "Proc----------2"

def proc3(a):
    while True:
        print "Proc----------3"



d1 = defer.Deferred()
d2 = defer.Deferred()
d3 = defer.Deferred()

d1.addCallback(proc1)
d2.addCallback(proc2)
d3.addCallback(proc3)

d1.callback('a')
d2.callback('a')
d3.callback('a')

reactor.run()

And for this snippet - each deferred callback is triggered one after the other and as far as the output is concerned there will only proc1 stdouts pouring indefinitely.

Please correct me if I am wrong folks. So basically what i want to understand and confirm is Deferred objects are triggered one after the other whereas the deferToThread are run parallely as by the name thread.

Answer 1

Please correct me if I am wrong folks. So basically what i want to understand and confirm is Deferred objects are triggered one after the other whereas the deferToThread are run parallely as by the name thread.

This isn't exactly right but it's sort of close. Here is the code that triggers your Deferreds:

d1.callback('a')
d2.callback('a')
d3.callback('a')

You triggered them one after the other. There's nothing particularly special or mysterious about this. It's just how Python works.

Deferreds don't have anything to do with threads. They don't automatically make code non-blocking or asynchronous or multi-threaded. They just keep a list of functions (you added to that list with the addCallback method) and then call the functions in that list (when you use the callback method).

Answer 2

Your understanding is OK.

Snippet 1 will run all the procedures at the same time (each in its own thread). The three procedures must be threadsafe.

Snippet 2 will run the procedures one after the other in the reactor thread. Only when one procedure completes will the reactor thread pass to the next deferred, and so begin executing the next procedure.

If one of the deferred is a part of twisted, say to access something over the network, when it blocks the reactor will move on to the next unblocked deferred it has, and eventually pickup the blocked one sometime after it unblocks.

You don't have to worry about one of your processes interfering with another (that is provided you don't have other code calling them or multiple reactors calling them)