using cookies with twisted.web.client

Question

I'm trying to make a web client application using twisted but having some trouble with cookies. Does anyone have an example I can look at?

Answer 1

While it's true that getPage doesn't easily allow direct access to the request or response headers (just one example of how getPage isn't a super awesome API), cookies are actually supported.

cookies = {cookies: tosend}
d = getPage(url, cookies=cookies)
def cbPage(result):
    print 'Look at my cookies:', cookies
d.addCallback(cbPage)

Any cookies in the dictionary when it is passed to getPage will be sent. Any new cookies the server sets in response to the request will be added to the dictionary.

You might have missed this feature when looking at getPage because the getPage signature doesn't have a cookies parameter anywhere in it! However, it does take **kwargs, and this is how cookies is supported: any extra arguments passed to getPage that it doesn't know about itself, it passes on to HTTPClientFactory.__init__. Take a look at that method's signature to see all of the things you can pass to getPage.

Answer 2

Turns out there is no easy way afaict The headers are stored in twisted.web.client.HTTPClientFactory but not available from twisted.web.client.getPage() which is the function designed for pulling back a web page. I ended up rewriting the function:

from twisted.web import client

def getPage(url, contextFactory=None, *args, **kwargs):
    fact = client._makeGetterFactory(
        url,
        HTTPClientFactory,
        contextFactory=contextFactory,
        *args, **kwargs)
    return fact.deferred.addCallback(lambda data: (data, fact.response_headers))