Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Works with urrlib.request but doesn't work with requests

I am trying to send a request wtih post method to an API, my code looks like the following one:

import urllib.request
import json

url         = "https://api.cloudflareclient.com/v0a745/reg"
referrer    = "e7b507ed-5256-4bfc-8f17-2652d3f0851f"
body        = {"referrer": referrer}
data        = json.dumps(body).encode('utf8')
headers     = {'User-Agent': 'okhttp/3.12.1'}
req         = urllib.request.Request(url, data, headers)
response    = urllib.request.urlopen(req)
status_code = response.getcode()
print (status_code)

Actually it works fine but i want to use "requests" library instead as it's more updated and more flexible with proxies with following code:

import requests
import json

url         = "https://api.cloudflareclient.com/v0a745/reg"
referrer    = "e7b507ed-5256-4bfc-8f17-2652d3f0851f"
data        = {"referrer": referrer}
headers     = {'User-Agent': 'okhttp/3.12.1'}
req         = requests.post(url, headers=headers, json=data)
status_code = req.status_code
print (status_code)

But it returns 403 status code, how can i fix it ?

Keep in mind that this API is open to everyone and you can just run the code with no worries.


EDIT-1: i have tried removing json.dumps(body).encode('utf8') or just .encode('utf8') from the second code by @tomasz-wojcik advice but i am still getting 403 while the first code still works!

EDIT-2: i tried requesting with postman that successfully made the request and returned 200 status code. postman generated the following python code:

import requests

url = "https://api.cloudflareclient.com/v0a745/reg"

payload = "{\"referrer\": \"e7b507ed-5256-4bfc-8f17-2652d3f0851f\"}"
headers = {
  'Content-Type': 'application/x-www-form-urlencoded',
  'User-Agent': 'okhttp/3.12.1',
  'Host': 'api.cloudflareclient.com'
}

response = requests.request("POST", url, headers=headers, data=payload)

status_code = response.status_code
print (status_code)

If you run the code outside of postman, it still returns 403 status code, i'm a litte confused, i am thinking that maybe "requests" library doesn't changing the user-agent in the second code.

EDIT-3: I have looked into it and found out that it works on python 2.7.16 but doesn't work on python 3.8.5!

EDIT-4: Some Developers are reporting that the second code works on python 3.6 too but the main thing is why it is working on other versions but not working on 3.8 or 3.7 ?

Python Versions that returned 403 status code(second code): 3.8.5 & 3.7

Python Versions that returned 200 status code(second code): 3.6 & 2.7.16

like image 585
Ashkan Avatar asked Aug 03 '20 18:08

Ashkan


1 Answers

The issue seems to be with how the host is handling ssl. Newer versions of requests uses certifi which in your case is having issues with the host server. I downgraded requests to an earlier version and it worked. (2.1.0). You can fix the version in your requirements.txt and it should work with any python version.

https://requests.readthedocs.io/en/master/user/advanced/#ca-certificates


Before version 2.16, Requests bundled a set of root CAs that it trusted, sourced from the Mozilla trust store. 
The certificates were only updated once for each Requests version. When certifi was not installed, this led to extremely out-of-date certificate bundles when using significantly older versions of Requests.

For the sake of security we recommend upgrading certifi frequently!
like image 110
ćarvaka Avatar answered Nov 03 '22 00:11

ćarvaka