Twisted: Creating a ThreadPool and then daemonizing leads to uninformative hangs

Question

I am developing a networked application in Twisted, part of which consists of a web interface written in Django.

I wish to use Twisted's WSGI server to host the web interface, and I've written a working "tap" plugin to allow me to use twistd.

When running the server with the -n flag (don't daemonize) everything works fine, but when this flag is removed the server doesn't respond to requests at all, and there are no messages logged (though the server is still running).

There is a bug on Twisted's Trac which seems to describe the problem exactly, and my plugin happens to be based on the code referenced in the ticket.

Unfortunately, the issue hasn't been fixed - and it was raised almost a year ago.

I have attempted to create a ThreadPoolService class, which extends Service and starts a given ThreadPool when startService is called:

class ThreadPoolService(service.Service):
    def __init__(self, pool):
        self.pool = pool

    def startService(self):
        super(ThreadPoolService, self).startService()
        self.pool.start()

    def stopService(self):
        super(ThreadPoolService, self).stopService()
        self.pool.stop()

However, Twisted doesn't seem to be calling the startService method at all. I think the problem is that with a "tap" plugin, the ServiceMaker can only return one service to be started - and any others belonging to the same application aren't started. Obviously, I am returning a TCPServer service which contains the WSGI root.

At this point, I've hit somewhat of a bit of a brick wall. Does anyone have any ideas as to how I can work round this issue?

Answer 1

Return a MultiService from your ServiceMaker; one that includes your ThreadPoolService as well as your main application service. The API for assembling such a thing is pretty straightforward:

multi = MultiService()
mine = TCPServer(...) # your existing application service
threads = ThreadPoolService()
mine.setServiceParent(multi)
threads.setServiceParent(multi)
return multi

Given that you've already found the ticket for dealing with this confusing issue within Twisted, I look forward to seeing your patch :).