Python-Requests, extract url parameters from a string

Question

I am using this awesome library called requests to maintain python 2 & 3 compatibility and simplify my application requests management.

I have a case where I need to parse a url and replace one of it's parameter. E.g:

http://example.com?param1=a&token=TOKEN_TO_REPLACE&param2=c 

And I want to get this:

http://example.com?param1=a&token=NEW_TOKEN&param2=c 

With the urllib I can achieve it this way:

from urllib.parse import urlparse from urllib.parse import parse_qs from urllib.parse import urlencode  url = 'http://example.com?param1=a&token=TOKEN_TO_REPLACE&param2=c'  o = urlparse(url) query = parse_qs(o.query) if query.get('token'):     query['token'] = ['NEW_TOKEN', ]     new_query = urlencode(query, doseq=True)     url.split('?')[0] + '?' + new_query  >>> http://example.com?param2=c&param1=a&token=NEW_TOKEN 

How can you achieve the same using the requests library?

Answer 1

You cannot use requests for this; the library builds such URLs if passed a Python structure for the parameters, but does not offer any tools to parse them. That's not a goal of the project.

Stick to the urllib.parse method to parse out the parameters. Once you have a dictionary or list of key-value tuples, just pass that to requests to build the URL again:

try:     # Python 3     from urllib.parse import urlparse, parse_qs except ImportError:     # Python 2     from urlparse import urlparse, parse_qs  o = urlparse(url) query = parse_qs(o.query) # extract the URL without query parameters url = o._replace(query=None).geturl()  if 'token' in query:     query['token'] = 'NEW_TOKEN'  requests.get(url, params=query) 

You can get both the urlparse and parse_qs functions in both Python 2 and 3, all you need to do is adjust the import location if you get an exception.

Demo on Python 3 (without the import exception guard) to demonstrate the URL having been built:

>>> from urllib.parse import urlparse, parse_qs >>> url = "http://httpbin.org/get?token=TOKEN_TO_REPLACE&param2=c" >>> o = urlparse(url) >>> query = parse_qs(o.query) >>> url = o._replace(query=None).geturl() >>> if 'token' in query: ...     query['token'] = 'NEW_TOKEN' ...  >>> response = requests.get(url, params=query) >>> print(response.text) {   "args": {     "param2": "c",      "token": "NEW_TOKEN"   },    "headers": {     "Accept": "*/*",      "Accept-Encoding": "gzip, deflate",      "Host": "httpbin.org",      "User-Agent": "python-requests/2.5.1 CPython/3.4.2 Darwin/14.1.0"   },    "origin": "188.29.165.245",    "url": "http://httpbin.org/get?token=NEW_TOKEN&param2=c" } 

Answer 2

Using requests only:

query = requests.utils.urlparse(url).query params = dict(x.split('=') for x in query.split('&'))  if 'token' in params:     params['token'] = 'NEW_TOKEN'  requests.get(url, params=params)