Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to prevent python requests from percent encoding my URLs?

I'm trying to GET an URL of the following format using requests.get() in python:

http://api.example.com/export/?format=json&key=site:dummy+type:example+group:wheel

#!/usr/local/bin/python  import requests  print(requests.__versiom__) url = 'http://api.example.com/export/' payload = {'format': 'json', 'key': 'site:dummy+type:example+group:wheel'} r = requests.get(url, params=payload) print(r.url) 

However, the URL gets percent encoded and I don't get the expected response.

2.2.1 http://api.example.com/export/?key=site%3Adummy%2Btype%3Aexample%2Bgroup%3Awheel&format=json 

This works if I pass the URL directly:

url = http://api.example.com/export/?format=json&key=site:dummy+type:example+group:wheel r = requests.get(url) 

Is there some way to pass the the parameters in their original form - without percent encoding?

Thanks!

like image 409
Satyen Rai Avatar asked May 06 '14 13:05

Satyen Rai


People also ask

Does Python request encode URL?

In Python, we can URL encode a query string using the urlib. parse module, which further contains a function urlencode() for encoding the query string in URL. The query string is simply a string of key-value pairs.

How do I encrypt a URL in Python?

You can encode multiple parameters at once using urllib. parse. urlencode() function. This is a convenience function which takes a dictionary of key value pairs or a sequence of two-element tuples and uses the quote_plus() function to encode every value.

Why does URL get encoded?

Why do we need to encode? URLs can only have certain characters from the standard 128 character ASCII set. Reserved characters that do not belong to this set must be encoded. This means that we need to encode these characters when passing into a URL.

Does Python requests use https?

Requests verifies SSL certificates for HTTPS requests, just like a web browser. SSL Certificates are small data files that digitally bind a cryptographic key to an organization's details.


1 Answers

It is not good solution but you can use directly string:

r = requests.get(url, params='format=json&key=site:dummy+type:example+group:wheel') 

BTW:

Code which convert payload to this string

payload = {     'format': 'json',      'key': 'site:dummy+type:example+group:wheel' }  payload_str = "&".join("%s=%s" % (k,v) for k,v in payload.items()) # 'format=json&key=site:dummy+type:example+group:wheel'  r = requests.get(url, params=payload_str) 

EDIT (2020):

You can also use urllib.parse.urlencode(...) with parameter safe=':+' to create string without converting chars :+ .

As I know requests also use urllib.parse.urlencode(...) for this but without safe=.

import requests import urllib.parse  payload = {     'format': 'json',      'key': 'site:dummy+type:example+group:wheel' }  payload_str = urllib.parse.urlencode(payload, safe=':+') # 'format=json&key=site:dummy+type:example+group:wheel'  url = 'https://httpbin.org/get'  r = requests.get(url, params=payload_str)  print(r.text) 

I used page https://httpbin.org/get to test it.

like image 192
furas Avatar answered Sep 29 '22 01:09

furas