Send "\r\n" symbols in python requests data

Question

Everything ok with curl:

curl -v "http://user:password@localhost/_control.html" -d $'data1=1\r\n'

I tried this way in python:

url = "http://localhost/_control.html"

payload = {'data1': '1\r\n'}

headers = {"Content-type": "application/x-www-form-urlencoded", "Accept": "text/plain"}

r = requests.post(url, data=payload, headers=headers, auth=('user', 'password'))

But it doesn't work. Content-length in this case is 13 instead of 9 (with curl request)

Is it possible to send same data (with \r\n at the end) using python requests?

Answer 1

The \r and \n characters are being URL-encoded, as they should be, because application/x-www-form-urlencoded data cannot contain those characters directly:

Non-alphanumeric characters are replaced by %HH, a percent sign and two hexadecimal digits representing the ASCII code of the character. Line breaks are represented as "CR LF" pairs (i.e., %0D%0A).

Logging the request sent from Python using this technique, we can see that it correctly sends these 13 bytes:

data1=1%0D%0A

In the case of curl, its manual page makes no mention of encoding of whatever you pass to -d/--data, so presumably you're expected to encode the string yourself before passing it to curl. We can confirm this with --trace-ascii -:

=> Send data, 9 bytes (0x9)
0000: data1=1

The \r\n pair doesn't show up clearly here, but we can infer that it's not encoded because of the byte count.

In short, the request you are sending with that curl command is not valid to begin with.