How to abort/cancel HTTP request in Python thread?

Question

I'm looking to abort/cancel an HTTP request in a Python thread. I have to stick with threads. I can't use asyncio or anything outside the standard library.

This code works fine with sockets:

"""Demo for Canceling IO by Closing the Socket

Works!

"""

import socket
import time

from concurrent import futures

start_time = time.time()

sock = socket.socket()


def read():
    "Read data with 10 second delay."
    sock.connect(('httpbin.org', 80))
    sock.sendall(b'GET /delay/10 HTTP/1.0\r\n\r\n')
    while True:
        data = sock.recv(1024)
        if not data:
            break
        print(data.decode(), end='')


with futures.ThreadPoolExecutor() as pool:
    future = pool.submit(read)
    futures.wait([future], timeout=5)
    sock.close()  # <-- Interrupt sock.recv(1024) in Thread:read().

end_time = time.time()
print(f'Duration: {end_time - start_time:.3f}')

# Duration is ~5s as expected.

Closing the socket in the main thread is used to interrupt the recv() in the executor pool's thread. The HTTP request should take 10 seconds but we only wait 5 seconds for it an then close the socket (effectively canceling the HTTP request/response).

Now I try using http.client:

"""Demo for Canceling IO in Threads with HTTP Client

Doesn't work!

"""

import time

from concurrent import futures

from http.client import HTTPConnection


def get(con, url):
    con.request('GET', url)
    response = con.getresponse()
    return response


start_time = time.time()

with futures.ThreadPoolExecutor() as executor:
    con = HTTPConnection('httpbin.org')
    future = executor.submit(get, con, '/delay/10')
    done, not_done = futures.wait([future], timeout=5)
    con.sock.close()

end_time = time.time()
print(f'Duration: {end_time - start_time:.3f}')

# Duration is ~10s unfortunately.

Unfortunately, the total duration is ~10 seconds here. Closing the socket does not interrupt the recv_into() in the client.

Seems like I am making some wrong assumptions. How do I interrupt the socket used in an http client from a separate thread?

Answer 1

What you describe is the intended well documented behavior:

Note close() releases the resource associated with a connection but does not necessarily close the connection immediately. If you want to close the connection in a timely fashion, call shutdown() before close().

Some further details regarding this behavior can still be found in CPython howto docs:

Strictly speaking, you're supposed to use shutdown on a socket before you close it. The shutdown is an advisory to the socket at the other end. Depending on the argument you pass it, it can mean "I'm not going to send anymore, but I'll still listen", or "I'm not listening, good riddance!". Most socket libraries, however, are so used to programmers neglecting to use this piece of etiquette that normally a close is the same as shutdown(); close(). So in most situations, an explicit shutdown is not needed.

One way to use shutdown effectively is in an HTTP-like exchange. The client sends a request and then does a shutdown(1). This tells the server "This client is done sending, but can still receive." The server can detect "EOF" by a receive of 0 bytes. It can assume it has the complete request. The server sends a reply. If the send completes successfully then, indeed, the client was still receiving.

Python takes the automatic shutdown a step further, and says that when a socket is garbage collected, it will automatically do a close if it's needed. But relying on this is a very bad habit. If your socket just disappears without doing a close, the socket at the other end may hang indefinitely, thinking you're just being slow. Please close your sockets when you're done.

Solution

Call shutdown before close.

Example

with futures.ThreadPoolExecutor() as executor:
    con = HTTPConnection('httpbin.org')
    future = executor.submit(get, con, '/delay/10')
    done, not_done = futures.wait([future], timeout=5)
    con.sock.shutdown()
    con.sock.close()

References

Python Socket objects - close: https://docs.python.org/3/library/socket.html#socket.socket.close

CPython Howto Sockets - disconnecting: https://github.com/python/cpython/blob/65460565df99fbda6a74b6bb4bf99affaaf8bd95/Doc/howto/sockets.rst#disconnecting