Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can I POST data with python requests lib with http-gzip or deflate compression?

I use the request-module of python 2.7 to post a bigger chunk of data to a service I can't change. Since the data is mostly text, it is large but would compress quite well. The server would accept gzip- or deflate-encoding, however I do not know how to instruct requests to do a POST and encode the data correctly automatically.

Is there a minimal example available, that shows how this is possible?

like image 692
AME Avatar asked Dec 06 '13 14:12

AME


People also ask

Can HTTP requests be compressed?

HTTP compression allows content to be compressed on the server before transmission to the client. For resources such as text this can significantly reduce the size of the response message, leading to reduced bandwidth requirements and download times.

When should you not use gzip?

If you take a file that is 1300 bytes and compress it to 800 bytes, it's still transmitted in that same 1500 byte packet regardless, so you've gained nothing. That being the case, you should restrict the gzip compression to files with a size greater than a single packet, 1400 bytes (1.4KB) is a safe value.

Is Python a gzip file?

Python's gzip module is the interface to GZip application. The gzip data compression algorithm itself is based on zlib module. The gzip module contains definition of GzipFile class along with its methods. It also caontains convenience function open(), compress() and decompress().


3 Answers

# Works if backend supports gzip

additional_headers['content-encoding'] = 'gzip'
request_body = zlib.compress(json.dumps(post_data))
r = requests.post('http://post.example.url', data=request_body, headers=additional_headers)
like image 148
KnightOrc Avatar answered Sep 18 '22 15:09

KnightOrc


I've tested the solution proposed by Robᵩ with some modifications and it works.

PSEUDOCODE (sorry I've extrapolated it from my code so I had to cut out some parts and haven't tested, anyway you can get your idea)

additional_headers['content-encoding'] = 'gzip'
s = StringIO.StringIO()
g = gzip.GzipFile(fileobj=s, mode='w')
g.write(json_body)
g.close()
gzipped_body = s.getvalue()
request_body = gzipped_body

r = requests.post(endpoint_url, data=request_body, headers=additional_headers)
like image 29
Marco Grassi Avatar answered Sep 20 '22 15:09

Marco Grassi


For python 3:

from io import BytesIO
import gzip

def zip_payload(payload: str) -> bytes:
    btsio = BytesIO()
    g = gzip.GzipFile(fileobj=btsio, mode='w')
    g.write(bytes(payload, 'utf8'))
    g.close()
    return btsio.getvalue()

headers = {
    'Content-Encoding': 'gzip'
}
zipped_payload = zip_payload(payload)
requests.post(url, zipped_payload, headers=headers)

like image 34
James D Avatar answered Sep 21 '22 15:09

James D