Using deferred objects inside Twisted applications

Question

I feel like I am not understanding some things in writing Twisted applications (.tac files). Using deferred objects in .py scripts is easy by just calling reactor.run() at the end, but I have not seen reactor.run() used in any twisted application sample code.

Can someone explain:

1. why reactor.run() is not called in twisted applications (or if this is an erroneous conclusion)
2. how I can use deferred objects inside a twisted application perhaps without calling reactor.run()
3. and general differences in writing twisted scripts vs applications.

Answer 1

1. Why is reactor.run() not called in example .tac files?

.tac files are meant to be loaded by the "twistd" command-line tool, which runs the reactor for you.

Running the reactor is something that's done once, by whatever bit of code is serving as the main-point for your program. Most Twisted code is effectively a plug-in of some kind, meant to operate in the context of a larger system.

In the particular case of .tac files, they are never meant to be run as stand-alone Python programs: their job is to produce an Application object (with an attached bunch of Service objects) that get started up when the reactor runs. It's important that the tac file itself not do much work on its own, because (for example) the Service implementations in question may need to separate code which needs to run privileged and unprivileged, which is an exacting process; if work is performed in the .tac itself, it may be haphazardly executed as the wrong user.

2. How I can use Deferreds inside a Twisted application, without calling reactor.run()?

Deferred is simply a mechanism for managing chains of callbacks. You don't need to call reactor.run(), or indeed, even have a reactor at all, to use them. For example:

>>> from twisted.internet.defer import Deferred
>>> d = Deferred()
>>> def hello(result):
...     print "'d' was fired:", result
...     return result + 3
... 
>>> d.addCallback(hello)
<Deferred at ...>
>>> print d
<Deferred at ...>
>>> d.callback(7)
'd' was fired: 7
>>> print d
<Deferred at ... current result: 10>

That said, many APIs that return a Deferred need the reactor to do some work in order to eventually call .callback() on it. For example, if you do ...

>>> from twisted.internet.task import deferLater
>>> from twisted.internet import reactor
>>> deferLater(reactor, 1.0, lambda: 20).addCallback(hello)
<Deferred at ...>
>>>

... you'll be sitting around forever waiting for that to fire unless somebody runs the reactor. Nothing will print until that happens.

But, if the reactor is already running - for example, if you were running this interactive example in python -m twisted.conch.stdio rather than python, you would see that Deferred get called back a second later, because that interactive prompt is already running the reactor.

3. What are the differences between Twisted scripts vs. applications?

These aren't really formally separated terms. Any Python script can potentially import code from Twisted and use it any way it wants to, so it's hard to say that any particular property applies to "scripts" except that they are computer programs :-).

If by Twisted Application you mean a .tac file or a plugin, the difference is that this kind of code is separated out into the part the builds the service (the code at the top level in your tac file or plugin) and the part that actually does the work (privilegedStartService/startService/stopService implementations of the services that said top-level code sets up). Also, code that runs in this context (i.e. is driven by twistd) does not need to run the reactor itself, since one will be set up and run by twistd itself. Such code must also therefore be careful to avoid importing twisted.internet.reactor, because twistd provides the ability to use different reactors (select, poll, epoll, kqueue, etc) and importing the reactor yourself before twistd has a chance to set it up will break this feature.