Python requests timeout not working properly

Question

I'm using the following code to check the average response time because i noticed it was very slow:

            t1 = time.time()
            response = requests.get(url, timeout=10, headers=headers)
            t2 = time.time()
            reqtimes += t2 - t1
            reqamount += 1
            print("Average response time:" + str(reqtimes/reqamount))

When i print the average response time after connecting to around 1000 different sites it tells me the average response time is 70 seconds. Why? my timeout is set to 10 !

Answer 1

The timeout is the maximum amount of time to allow for any response from the server^*. If the timeout is 10 seconds, and the server returns a 100 byte file, a single byte at a time every 9 seconds, then you'll never time out... but you'll have a very long wait for the response to complete (total elapsed time will be 900 seconds). eg: Even a single request with a timeout of 10 seconds will have an average response time of 15 minutes).

It's worth noting that response objects already have an .elapsed attribute which gives you a timedelta object you can use to avoid using time.time() and performing calculations yourself.

^*Paraphrased from http://docs.python-requests.org/en/master/user/quickstart/#timeouts:

timeout is not a time limit on the entire response download; rather, an exception is raised if the server has not issued a response for timeout seconds (more precisely, if no bytes have been received on the underlying socket for timeout seconds). If no timeout is specified explicitly, requests do not time out.