Python requests sometimes freezes

Question

I have a Python program which sends several (about 5-6) long poll requests in parallel using different threads for each poll via requests package. And I realized that some of my threads sometimes just freeze. When this happens, the server I am sending the request does not receive the request. Also I set a timeout on the request and it does not work.

try:
    print("This line prints")
    response = requests.head(poll_request_url, timeout=180)
    print("This line does not print when freeze occurs")
except ReadTimeout:
    print("Request exception.")
except RequestException as e:
    print("Request exception.")
except Exception:
    print("Unknown exception.")
print("This line does not print either when freeze occurs.")

I am doing this on Raspberry Pi 2 hardware with Raspbian OS.

I used this same program without a problem when I was using Python 2.7. Recently I switched to Python 3.5. I tested using both requests versions with 2.8.1 and 2.9.1.

This problem does not occur very frequently but happens 2-3 times per day on different threads.

What might be the problem? How can I debug this?

Edit: The problem is solved by updating the Linux kernel.

Answer 1

According to the docs:

http://docs.python-requests.org/en/master/user/quickstart/#timeouts

It should be throwing a Timeout exception, when the timeout happens. That would mean that the line:

print("This line does not print when freeze occurs")

Would never be called it a timeout actually happens.

Are you catching the exception? Or any other exception? It might be that it's timing out fine, but you just aren't seeing this. Maybe try something like this:

try:
    response = requests.head(poll_request_url, timeout=180)
except requests.exceptions.Timeout:
    print("Timeout occurred")

So you can see if that's what is going on.

EDIT: possibly it's the "connect" step that's not timing out correctly. It may be the large timeout value for the "connect" step is messing it up somehow. Perhaps trying having a shorter timeout for that (as mentioned here):

http://docs.python-requests.org/en/master/user/advanced/#timeouts

e.g.

response = requests.head(poll_request_url, timeout=(3, 180))

Failing that it might be some sort of DNS lookup issue? Maybe see if hardcoding the IPs presents the same problem?

Answer 2

Solved my problem using timers (from threading import Timer). If no result next 10 seconds - repeat, if no result next 10 seconds - print 'Error' and go on. You can't monitor timer status with if statement if request freezes, but you can do it through while loop, adding time if result is ok (Python: Run code every n seconds and restart timer on condition).