unable to decode Python web request

Question

I am trying to make web request to my trading account . Python is unable to decode the web request. Web request is successful with code - 200.

Here is the code below

import requests

headers = {
    'accept-encoding': 'gzip, deflate, br',
    'accept-language': 'en-US,en;q=0.9',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36',
    'x-kite-version': '1.2.1',
    'accept': 'application/json, text/plain, */*',
    'referer': 'https://kite.zerodha.com/orders',
    'authority': 'kite.zerodha.com',
    'cookie': '__cfduid=db8fb54c76c53442fb672dee32ed58aeb1521962031; _ga=GA1.2.1516103745.1522000590; _gid=GA1.2.581693731.1522462921; kfsession=CfawFIZq2T6SghlCd8FZegqFjNIKCYuO; public_token=7FyfBbbxhiRRUso3425TViK2VmVszMCK; user_id=XE4670',
    'x-csrftoken': '7FyfBbbxhiRRUso3425TViK2VmVszMCK',
}

response = requests.get('https://kite.zerodha.com/api/orders', headers=headers)


x=str(response.content.decode("utf-8") )

---

b"1X\x14\x00 \xfe\xa7\x9b\xd3\xca\xbd9-\x12\x83\xbfULS1\x1d8\x9d\x0e\xd4\xcf\xbd\xb8\xd1\xbd4\xc0\x00\x13~\x94}\xe4\x81\xa4\x90P\x1cfs\xcd\x1e\xaeG\x9b},m\xbd\t\x84L1\xde\xa8e\x8a\xf1h\x0e\x0c)\x1a\x12\xfb\x06z\xec\x18\xe4r\xa1\x1c\x11\xe8 \xbcO\xec\xe2|\xa6\x90\xa9\xdf\xf2\xe1\xfa\xf3\x1e\x04\x0e\xa2\x8d\x0e\xc4\tw\xeb\xd9\xba\n\xf1H'l\xeb>\x08\x85L\r\x0cY\xf8\x81D;\x92!o\xfd\xbd\xe3u>3\x10\xe1\x8c;\xb8\x9e\xceA\xae\x0exX\xc9\x19s\xeb\xe5r~1\x98\xed0\xb8\xdc\xb4\x17:\x14\x96xAn\xb9\xf0\xce\xf2l\\xa6G?5O\x9b\xf3\xc1\\x1f\x0f\x8fs\x1b/\x17\x1a\x0c[ySAX\x1d'\xe7\xbb\nx\xacR~\xbb\x9f\xe0\x8c?s\xc0\x8f\xe0\x97\xff\xde'\xc7#\x8f\x97\xaf\xaa%\xf2\xf9\xfaC|\xcf\t\xf3\xeb\xaa\xdcs\xcc\xf5\xa3RM\xbaOY\xf5\x9fe\xfc\x07\xff\x01"

Unable to decode this. Tried unicode- utf 8 and various codes available on the stakoverflow but its failed.

Answer 1

According to the response.headers (that you did not provide, but that are easily recoverable by running your code), the response is encoded using Brotli compression (Content-Encoding': 'br'). You can decompress it with brotlipy:

import brotli
brotli.decompress(response.content)
#b'{"status":"success","data":[{"placed_by":"XE4670","order_id":"180331000000385",  
#"exchange_order_id":null,"parent_order_id":null,"status":"REJECTED",
#"status_message":"ADAPTER is down","order_timestamp":"2018-03-31 07:59:42", 
#"exchange_update_timestamp":null,...}

Now, it's JSON, as promised ('Content-Type': 'application/json').

Answer 2

If the server only returns brotli compressed response, response should be decompressed then it will be ready to use.

Fortunately, since v2.26.0 update, requests library supports Brotli compression if either the brotli or brotlicffi package is installed. So, if the response encoding is br, request library will automatically handle it and decompress it.

First;

pip install brotli,

then;

import requests

r = requests.get('some_url')
r.json()