How can I view/get scrapy POST/GET request headers

Question

I'd like to know if there's any method to view headers that are being sent when scrapy issues a POST/GET request within live logs, shell or by any other similar means. Thanks!

Answer 1

Both Response and Request objects will have their headers available via the .headers attribute.

Headers for both objects are modified via the Middleware between the Downloader and the Engine (see Scrapy Architecture). If you create a new Request object, it won't have any headers until it's passed through the Middleware that assigns them.

To view the request object, as it will be sent out, you will need to create a Middleware, put it closer to the Downloader than any other header-altering Middleware, and check the request.headers attribute at that time.

Alternatively, you can check out the headers of the request which returned a Response to the Spider by viewing the response.request.headers. This may not be the Request object you sent out though, but the one that the resulted in the Response object that was returned (for instance, redirects/retries result in the originally dispatched Request object being different than the Request object in response.request). Of course, this requires a Response object to have been returned to the spider, so it won't work for any Request that didn't generate a response (eg. DNS lookup error), or any Response that gets ignored or dropped via Middleware (eg. HTTP Status 400).