Why does Twisted think I'm calling request.finish() twice when I am not?

Question

This is an annoying problem I am having with Twisted.web. Basically, I have a class that inherits from twisted.web.resource.Resource and adds some default stuff to Mako templates:

from twisted.web.resource import Resource
from mako.lookup import TemplateLookup
from project.session import SessionData
from project.security import make_nonce


class Page(Resource):

    template = ""

    def display(self, request, **kwargs):
        session = SessionData(request.getSession())

        if self.template:
            templates = TemplateLookup(directories=['templates'])
            template = templates.get_template(self.template)
            return template.render(user=session.user,
                                   info=session.info,
                                   current_path=request.path,
                                   nonce=make_nonce(session),
                                   **kwargs)
        else:
            return ""

Then, and I have narrowed the problem down to this small class (which I tested), I write a resource which inherits from Page:

class Test(pages.Page):
    def render_GET(self, request):
        return "<form method='post'><input type='submit'></form>"
    def render_POST(self, request):
        request.redirect("/test")
        request.finish()

I'd like to note that, in every other case, if request.finish() isn't the last line in a function, then I return immediately after it.

Anyways, I add this class to the site at /test and when I navigate there, I get a submit button. I click the submit button, and in the console I get:

C:\Python26\lib\site-packages\twisted\web\server.py:200: UserWarning: Warning! request.finish called twice.
  self.finish()

But, I get this ONLY the first time I submit the page. Every other time, it's fine. I would just ignore this, but it's been nagging at me, and I can't for the life of me figure out why it's doing this at all, and why only the first time the page is submitted. I can't seem to find anything online, and even dropping print statements and tracebacks in the request.finish() code didn't reveal anything.

edit

This morning I tried adding a second request.finish() line to the resource, and it still only gave me the error one time. I suppose it will only warn about it in a resource once -- maybe per run of the program, or per session, I'm not sure. In any case, I changed it to:

class Test(pages.Page):
    def render_GET(self, request):
        return "<form method='post'><input type='submit'></form>"
    def render_POST(self, request):
        request.redirect("/test")
        request.finish()
        request.finish()

and just got two messages, one time. I still have no idea why I can't redirect the request without it saying I finished it twice (because I can't redirect without request.finish()).

Answer 1

Short Answer

---

It has to be:

request.redirect("/test")
request.finish()
return twisted.web.server.NOT_DONE_YET

Long Answer

---

I decided to go sifting through some Twisted source code. I first added a traceback to the area that prints the error if request.finish() is called twice:

def finish(self):
    import traceback #here
    """
    Indicate that all response data has been written to this L{Request}.
    """
    if self._disconnected:
        raise RuntimeError(
            "Request.finish called on a request after its connection was lost; "
            "use Request.notifyFinish to keep track of this.")
    if self.finished:
        warnings.warn("Warning! request.finish called twice.", stacklevel=2)
        traceback.print_stack() #here
        return
    #....

...
  File "C:\Python26\lib\site-packages\twisted\web\server.py", line 200, in render
    self.finish()
  File "C:\Python26\lib\site-packages\twisted\web\http.py", line 904, in finish
    traceback.print_stack()

I went in and checked out render in twisted.web.server and found this:

    if body == NOT_DONE_YET:
        return
    if type(body) is not types.StringType:
        body = resource.ErrorPage(
            http.INTERNAL_SERVER_ERROR,
            "Request did not return a string",
            "Request: " + html.PRE(reflect.safe_repr(self)) + "<br />" +
            "Resource: " + html.PRE(reflect.safe_repr(resrc)) + "<br />" +
            "Value: " + html.PRE(reflect.safe_repr(body))).render(self)

    if self.method == "HEAD":
        if len(body) > 0:
            # This is a Bad Thing (RFC 2616, 9.4)
            log.msg("Warning: HEAD request %s for resource %s is"
                    " returning a message body."
                    "  I think I'll eat it."
                    % (self, resrc))
            self.setHeader('content-length', str(len(body)))
        self.write('')
    else:
        self.setHeader('content-length', str(len(body)))
        self.write(body)
    self.finish()

body is the result of rendering a resource, so once body is populated, in the example case given in my question, finish has already been called on this request object (since self is passed from this method to the resource's render method).

From looking at this code it becomes apparent that by returning NOT_DONE_YET I would avoid the warning.

I could have also changed the last line of that method to:

if not self.finished:
    self.finish()

but, in the interest of not modifying the library, the short answer is:

after calling request.redirect() you must call request.finish() and then return twisted.web.server.NOT_DONE_YET

More

---

I found some documentation about this. It isn't related to redirecting a request, but instead rendering a resource, using request.write(). It says to call request.finish() and then return NOT_DONE_YET. From looking at the code in render() I can see why that is the case.