Python - HTTP module cannot parse response if the server answers before the PUT is complete

Question

I'm using the requests (which uses urllib3 and the Python http module under the hood) library to upload a file from a Python script. My backend starts by inspecting the headers of the request and if it doesn't comply with the needed prerequisites, it stops the request right away and respond with a valid 400 response.

This behavior works fine in Postman, or with Curl; i.e. the client is able to parse the 400 response even though it hasn't completed the upload and the server answers prematurely. However, while doing so in Python with requests/urllib3, the library is unable to process the backend response :

Traceback (most recent call last):
  File "C:\Users\Neumann\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\urllib3\connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "C:\Users\Neumann\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\urllib3\connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1776.0_x64__qbz5n2kfra8p0\lib\http\client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1776.0_x64__qbz5n2kfra8p0\lib\http\client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1776.0_x64__qbz5n2kfra8p0\lib\http\client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1776.0_x64__qbz5n2kfra8p0\lib\http\client.py", line 1049, in _send_output
    self.send(chunk)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1776.0_x64__qbz5n2kfra8p0\lib\http\client.py", line 971, in send
    self.sock.sendall(data)
ConnectionResetError: [WinError 10054] Une connexion existante a dû être fermée par l’hôte distant

Because the server answers before the transfer is complete, it mistakenly considers that the connection has been aborted, even though the server DOES return a valid response.

Is there a way to avoid this and parse the response nonetheless ?

Steps to reproduce the issue :

• Download minIO : https://min.io/download#/
• Run minIO :

export MINIO_ACCESS_KEY=<access_key>
export MINIO_SECRET_KEY=<secret_key>
.\minio.exe server <data folder>

• Run the following script :

import os
import sys
import requests
from requests_toolbelt.multipart.encoder import MultipartEncoder

def fatal(msg):
    print(msg)
    sys.exit(1)

def upload_file():
    mp_encoder = MultipartEncoder(fields={'file': (open('E:/Downloads/kek.mp3', 'rb'))})
    headers = { "Authorization": "invalid" }

    print('Uploading file with headers : ' + str(headers))

    upload_endpoint = 'http://localhost:9000/mybucket/myobject'
    try:
        r = requests.put(upload_endpoint, headers=headers, data=mp_encoder, verify=False)
    except requests.exceptions.ConnectionError as e:
        print(e.status)
        for property, value in vars(e).items():
            print(property, ":", value)
        fatal(str(e))
    
    if r.status_code != 201:
        for property, value in vars(r).items():
            print(property, ":", value)
        fatal('Error while uploading file. Status ' + str(r.status_code))
    print('Upload successfully completed')

if __name__ == "__main__":
    upload_file()

If you change the request line with this, it will work (i.e. the server returns 400 and the client is able to parse it) :

r = requests.put(upload_endpoint, headers=headers, data='a string', verify=False)

EDIT : I updated the traceback and changed the question title to reflect the fact that it's neither requests or urllib3 fault, but the Python http module that is used by both of them.

Answer 1

This problem should be fixed in urllib3 v1.26.0. What version are you running?

The problem is that the server closes the connection after it responds with 400, so the socket is closed when urllib3 tries to keep sending data to it. So it isn't really mistakenly thinking that the connection is closed, it just mishandles that situation.

Your example code works fine on my machine with urllib3==1.26.0 . But I notice that you get a different exception on your Windows machine, so it might be that the fix doesn't work. In that case, I would just catch the exception and file a bug report to the maintainers of urllib3.