Asynchronous WSGI with Twisted

Question

I'm building a web interface for a twisted application and would like to use WSGI rather than twisted.web directly (since the rest of the website is WSGI and I already have a substantial WSGI codebase).

The Twisted documentation page I found about WSGIResource (http://twistedmatrix.com/documents/current/web/howto/web-in-60/wsgi.html) states: Like any other WSGI container, you can't do anything asynchronous in your WSGI applications, even though this is a Twisted WSGI container.

Does this have to be true? Is there some less-than-hacky way of doing twisted.web style asynchronous web request handling in WSGI - perhaps as part of another free software project? Supposing there isn't, my plan is to have WSGI threads do their asynchronous work in the reactor thread and block by polling until the data is available. It's not pretty.

If there's a reasonably uncomplicated way of asynchronously handling WSGI requests in twisted I'd love to hear it.

Answer 1

Why do you want to use WSGI and do asynchronous things? The benefit of WSGI is that you can deploy your application on any WSGI container. If you start using Twisted APIs to do asynchronous things, then you can only deploy your application in Twisted's WSGI container.

You should probably just use Twisted Web without WSGI for your asynchronous code.

Answer 2

In principle, WSGI is not intrinsically incompatible with asynchronous program design; in fact, PEP 333 goes to some considerable length to specify how servers, applications and middleware must behave to support that kind of thing.

At the heart of this is returning an iterator to the container. Every time an asynchronous wsgi app_iter is invoked, it would check on all of its pending asyncronous tasks (database connections, etcetera) and if any of them have data, the app_iter yields some data; otherwise it yields an empty string. To support this, a wsgi container would need to keep track of all of the in-flight requests, and iterate each of them in turn to get more data, in addition to servicing any other deferred work that it is responsible for.

In principle, very few wsgi apps or frameworks actually do this. almost invariably, wsgi frameworks block for all sorts of reasons; reading files from disk or loading data from a database for any reason at all (Most ORM's make this a tough problem to prevent.) Twisted's wsgi container operates under the assumption that since some wsgi apps block, that perhaps any wsgi app may block, and therefore always runs them in a thread.

There are two things you can do; either explore twisted's own web framework, which is fairly solid; or consider creating a wsgi wrapper for twisted outside of twisted's own container. Making sure the wsgi app is actually asyncronous is certainly a precondition of the latter, but wsgi itself is pretty simple, a thin wrapper over http, and so it should be easy enough.