Python 3 - requests library - iter_lines - Handling possible server stalls when reading chunk of streamed data

Question

I need to find out a way to smoothly manage server stalls when I have been reading data from it. I wrote the piece of code below:

def listener():    
    resp = requests.get(someurl, stream=True)
    if resp.status_code == 200:
        for line in resp.iter_lines():            
            if line:
                do_something_with_the_line
                print(result)

price_thread = threading.Thread(target=listener, name="StreamingThread", args=[])
trade_thread.start()

The code works well until a stall of the server occur (the API provider suggests a stall occur when no "lines" are received in 10 seconds).

How can I implement this in my code? In other words I would try to recall the listener method without exiting the price_thread thread when a stall occur.

Thanks in advance

Answer 1

You can use the timeout property to ensure that your connection breaks off after no data is received in the specified amount of time, and then respawn the connection:

def listener():
    while True:
        try:
            resp = requests.get(someurl, stream=True, timeout=10)
            if resp.status_code == 200:
                for line in resp.iter_lines():
                    if line:
                        # do_something_with_the_line
                        pass
            elif resp.status_code == 429:
                    print("Too many reconnects, exiting.")
                    break
            else:
                print("Unhandled status `{}` retreived, exiting.".format(resp.status_code))
                break
        except requests.exceptions.Timeout:
            pass  # we'll ignore timeout errors and reconnect
        except requests.exceptions.RequestException as e:
            print("Request exception `{}`, exiting".format(e))
            break

You may also add a reconnect counter instead of having the while True loop indefinitely.