Python requests - GET/POST with data over SSL truncates the response

Question

For example take this app.py and run it with python3 -m flask run --cert=adhoc.

from flask import Flask
app = Flask(__name__)

@app.route('/', methods=["GET", "POST"])
def hello_world():
    return {"access_token": "aa" * 50000}

How could sending data truncate the response?

>>> import requests
>>> len(requests.post('https://127.0.0.1:5000/', verify=False).content)
100020
>>> len(requests.post('https://127.0.0.1:5000/', data={'a':'b'}, verify=False).content)
81920

PS 1: It works as expected without SSL;

PS 2: GET produces the same behaviour;

PS 3: Curl produces the correct result:

$ curl -s -k -X POST https://127.0.0.1:5000/ -H 'Content-Type: application/json' -d '{"a":"b"}' | wc -m
100020

PS 4: I reported this as a bug on requests' github.

Answer 1

This is a weird one, and I confess I am not entirely sure if my answer is really correct in every detail. But it seems you might need to report this to Flask rather than requests...

@app.route('/', methods=["GET", "POST"])
def hello_world():
    print(f"hello_world: content_type: {request.content_type}, data: {request.data}")
    return {"access_token": "aa" * 50000}

This code responds as expected to both

>>> len(requests.post('https://127.0.0.1:5000/', data={"a":"b"}, verify=False).content)

and

>>> len(requests.post('https://127.0.0.1:5000/', verify=False).content)

Now comment out the print(), and the data=... request gets an exception:

raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

I suspect that Flask will read the data from the request only on demand. But with SSL, reading data will affect encryption (e.g. via cipher block chaining). So sending a response before fully reading the request leads to encoding problems.

So read all the data you get from the request before you send an answer.