Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

POST request works in Postman, but not in Python Requests (200 response with robot detection)

I have a POST request that works perfectly with both Postman an cURL (it returns a JSON blob of data). However, when I perform the exact same request with Python's Requests library, I get a 200 success response, but instead of my JSON blob, I get this:

<html>
<head>
<META NAME="robots" CONTENT="noindex,nofollow">
<script src="/_Incapsula_Resource?SWJIYLWA=5074a744e2e3d891814e9a2dace20bd4,719d34d31c8e3a6e6fffd425f7e032f3">
</script>
<body>
</body></html>

I've used HTTP request bins to verify that the request from Postman/cURL is exactly the same as the one from Python Requests.

Here is my Postman request in cURL:

curl -X POST \
  https:/someurl/bla/bla \
  -H 'Content-Type: application/json' \
  -H 'Postman-Token: 2488e914-531e-4ac7-ae8d-8490b2242396' \
  -H 'Referer: https://www.host.com/bla/bla/' \
  -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:65.0) Gecko/20100101 Firefox/65.0' \
  -H 'cache-control: no-cache' \
  -d '{"json1":"blabla","etc":"etc"}'

...and here is my Python code:

payload = {
      "json1": "blabla",
      "etc": "etc",
    }

    headers = {
        'Host': 'www.host.com',
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36',
        'Accept': 'application/json, text/javascript, */*; q=0.01',
        'Accept-Language': 'en-US,en;q=0.5',
        'Accept-Encoding': 'gzip, deflate, br',
        'Referer': 'https://www.host.com/bla/bla/', 
        'Content-Type':'application/json',
        'X-Requested-With': 'XMLHttpRequest',
        'Connection': 'keep-alive',
        'Origin': 'https://www.host.com',
    }

    s = requests.Session()
    response_raw = s.post(url, json=payload, headers=headers)
    print(response_raw)
    print(response_raw.text)

I have verified that the payload and headers are correct and valid. Any help would be much appreciated; thanks!

like image 402
woodenstick Avatar asked Feb 26 '19 05:02

woodenstick


People also ask

Does Postman work with Python?

As an example, you can use Postman to generate Python code for the call that shuffles six decks, then have the Python code use that generated deck_id to draw two cards from the deck.

How do you get a response from a Python request?

When one makes a request to a URI, it returns a response. This Response object in terms of python is returned by requests. method(), method being – get, post, put, etc.

How does Python requests post work?

Understanding the Python requests POST FunctionAn HTTP POST request is used to send data to a server, where data are shared via the body of a request. In the request. post() function, data are sent with the data parameter, which accepts a dictionary, a list of tuples, bytes or a file object.


1 Answers

You are getting a 200 success response but not JSON data in the response.
This means that is just a response object. It contains only response code
to extract blob information from the response, convert response object to json
simply json_resp = response_raw.json()
This json_resp contains your actual response details.

like image 132
venkat Avatar answered Sep 29 '22 16:09

venkat