Python requests view before sending

Question

Can somebody please provide me with a way to see a request that I have generated before sending it to a server, here is the code:

import requests
import urllib2
import logging

from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

url = "https://10.1.1.254:4081/admin/api/jsonrpc/"
session = requests.Session()

data = {
    "jsonrpc": "2.0", "id": 1, "method": "Session.login",
    "params": {"userName": "test", "password":"test123"}
}
r = session.post(url, json=data, verify=False)

data = {"jsonrpc": "2.0", "id":3, "method":"Session.logout"}
r = session.post(url, json=data, verify=False)

So what I would like is to get that request sent with session.post, before Python sends it.

Answer 1

You have three options here:

• by using a prepared request,
• inspecting the request after it has been sent,
• you can enable logging in urllib3 as well as enable debugging in the http.client library.

They give you different levels of control over what is output.

Prepared requests

You can prepare a request, which gives you a new object with all the data set up in final form. You can look at the requests.PreparedRequest API docs for the full list of attributes.

Here is a sample function that simply formats that data as if it were sent over the network (except it doesn't actually use CRLF line separators, and replacing binary content with a placeholder string):

def format_prepped_request(prepped, encoding=None):
    # prepped has .method, .path_url, .headers and .body attribute to view the request
    encoding = encoding or requests.utils.get_encoding_from_headers(prepped.headers)
    body = prepped.body.decode(encoding) if encoding else '<binary data>' 
    headers = '\n'.join(['{}: {}'.format(*hv) for hv in prepped.headers.items()])
    return f"""\
{prepped.method} {prepped.path_url} HTTP/1.1
{headers}

{body}"""

Use a prepareded request instead of using session.[httpmethod](), and instead use session.send() to send out the prepared request after you've inspected it. Put the HTTP method as the first argument of Request() instead (uppercased). So session.post(...) becomes Request('POST', ...). All other arguments, except for verify are moved to the Request() call:

url = "https://10.1.1.254:4081/admin/api/jsonrpc/"
session = requests.Session()

data = {
    "jsonrpc": "2.0", "id":1, "method": "Session.login",
    "params": {"userName": "test", "password":"test123"}
}

request = requests.Request('POST', url, json=data)
prepped = session.prepare_request(request)

print("Sending request:")
print(format_prepped_request(prepped, 'utf8'))
print()
r = session.send(prepped, verify=False)

data = {"jsonrpc": "2.0", "id":3, "method":"Session.logout"}
request = requests.Request('POST', url, json=data)
prepped = session.prepare_request(request)

print("Sending request:")
print(format_prepped_request(prepped, 'utf8'))
print()
r = session.send(prepped, verify=False)

For your sample requests, this outputs:

Sending request:
POST /admin/api/jsonrpc/ HTTP/1.1
User-Agent: python-requests/2.22.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Content-Length: 109
Content-Type: application/json

{"jsonrpc": "2.0", "id": 1, "method": "Session.login", "params": {"userName": "test", "password": "test123"}}

Sending request:
POST /admin/api/jsonrpc/ HTTP/1.1
User-Agent: python-requests/2.22.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Content-Length: 55
Content-Type: application/json

{"jsonrpc": "2.0", "id": 3, "method": "Session.logout"}

Inspecting the request after sending

The response object references the prepared request object used to make the request, so you can print this after sending the request:

# ...
r = session.post(url, json=data, verify=False)
prepped = r.request
print("Request that was sent:")
print(format_prepped_request(prepped, 'utf8'))
print()

This isn't quite the same of course, the request has now already been sent. On the other hand, you don't need to use prepared requests directly, session.post() went through the steps to produce that object itself.

Do take into account that if your original request led to a redirect, that the final request may differ. Look at the response.history attribute to access any preceding responses (each with their own request attached).

Logging

Using the logging module, as well as rigging up the http.client.HTTPConnection class to log its debug output to the same framework, you can get debug-level information on requests too. I'm using my answer to Log all requests from the python-requests module here:

import logging

# function definition from https://stackoverflow.com/a/16337639/100297
# configures http.client to log rather than print
httpclient_logging_patch()
logging.basicConfig(level=logging.DEBUG)

at which point you'll see requests appear in logging output:

DEBUG:http.client:send: b'POST /admin/api/jsonrpc/ HTTP/1.1\r\nHost: 10.1.1.254:4081\r\nUser-Agent: python-requests/2.22.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 109\r\nContent-Type: application/json\r\n\r\n'
DEBUG:http.client:send: b'{"jsonrpc": "2.0", "id": 1, "method": "Session.login", "params": {"userName": "test", "password": "test123"}}'
DEBUG:http.client:reply: 'HTTP/1.1 200 OK\r\n'
DEBUG:http.client:header: Date: Tue, 04 Feb 2020 13:41:52 GMT
DEBUG:http.client:header: Content-Type: application/json
DEBUG:http.client:header: Content-Length: 574
DEBUG:http.client:header: Connection: keep-alive
DEBUG:http.client:header: Server: gunicorn/19.9.0
DEBUG:http.client:header: Access-Control-Allow-Origin: *
DEBUG:http.client:header: Access-Control-Allow-Credentials: true
DEBUG:urllib3.connectionpool:https://10.1.1.254:4081 "POST /admin/api/jsonrpc/ HTTP/1.1" 200 574
DEBUG:http.client:send: b'POST /admin/api/jsonrpc/ HTTP/1.1\r\nHost: 10.1.1.254:4081\r\nUser-Agent: python-requests/2.22.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 55\r\nContent-Type: application/json\r\n\r\n'
DEBUG:http.client:send: b'{"jsonrpc": "2.0", "id": 3, "method": "Session.logout"}'
DEBUG:http.client:reply: 'HTTP/1.1 200 OK\r\n'
DEBUG:http.client:header: Date: Tue, 04 Feb 2020 13:43:56 GMT
DEBUG:http.client:header: Content-Type: application/json
DEBUG:http.client:header: Content-Length: 574
DEBUG:http.client:header: Connection: keep-alive
DEBUG:http.client:header: Server: gunicorn/19.9.0
DEBUG:http.client:header: Access-Control-Allow-Origin: *
DEBUG:http.client:header: Access-Control-Allow-Credentials: true
DEBUG:urllib3.connectionpool:https://10.1.1.254:4081 "POST /admin/api/jsonrpc/ HTTP/1.1" 200 574

This isn't as flexible as using prepared requests but would work with existing code-bases. It also outputs data for the response, but you can do the same with the response object that requests gives you and print that out like we did with prepared requests above.