How can I see the entire HTTP request that's being sent by my Python application?

Question

In my case, I'm using the requests library to call PayPal's API over HTTPS. Unfortunately, I'm getting an error from PayPal, and PayPal support cannot figure out what the error is or what's causing it. They want me to "Please provide the entire request, headers included".

How can I do that?

Answer 1

A simple method: enable logging in recent versions of Requests (1.x and higher.)

Requests uses the http.client and logging module configuration to control logging verbosity, as described here.

Demonstration

Code excerpted from the linked documentation:

import requests import logging  # These two lines enable debugging at httplib level (requests->urllib3->http.client) # You will see the REQUEST, including HEADERS and DATA, and RESPONSE with HEADERS but without DATA. # The only thing missing will be the response.body which is not logged. try:     import http.client as http_client except ImportError:     # Python 2     import httplib as http_client http_client.HTTPConnection.debuglevel = 1  # You must initialize logging, otherwise you'll not see debug output. logging.basicConfig() logging.getLogger().setLevel(logging.DEBUG) requests_log = logging.getLogger("requests.packages.urllib3") requests_log.setLevel(logging.DEBUG) requests_log.propagate = True  requests.get('https://httpbin.org/headers') 

Example Output

$ python requests-logging.py  INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): httpbin.org send: 'GET /headers HTTP/1.1\r\nHost: httpbin.org\r\nAccept-Encoding: gzip, deflate, compress\r\nAccept: */*\r\nUser-Agent: python-requests/1.2.0 CPython/2.7.3 Linux/3.2.0-48-generic\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Content-Type: application/json header: Date: Sat, 29 Jun 2013 11:19:34 GMT header: Server: gunicorn/0.17.4 header: Content-Length: 226 header: Connection: keep-alive DEBUG:requests.packages.urllib3.connectionpool:"GET /headers HTTP/1.1" 200 226 

Answer 2

r = requests.get('https://api.github.com', auth=('user', 'pass')) 

r is a response. It has a request attribute which has the information you need.

r.request.allow_redirects  r.request.headers          r.request.register_hook r.request.auth             r.request.hooks            r.request.response r.request.cert             r.request.method           r.request.send r.request.config           r.request.params           r.request.sent r.request.cookies          r.request.path_url         r.request.session r.request.data             r.request.prefetch         r.request.timeout r.request.deregister_hook  r.request.proxies          r.request.url r.request.files            r.request.redirect         r.request.verify 

r.request.headers gives the headers:

{'Accept': '*/*',  'Accept-Encoding': 'identity, deflate, compress, gzip',  'Authorization': u'Basic dXNlcjpwYXNz',  'User-Agent': 'python-requests/0.12.1'} 

Then r.request.data has the body as a mapping. You can convert this with urllib.urlencode if they prefer:

import urllib b = r.request.data encoded_body = urllib.urlencode(b) 

depending on the type of the response the .data-attribute may be missing and a .body-attribute be there instead.