I would like to add a retry mechanism to python request library, so scripts that are using it will retry for non fatal errors.
At this moment I do consider three kind of errors to be recoverable:
At the first stage I do want to retry specified 5xx requests every minute.
I want to be able to add this functionality transparently, without having to manually implement recovery for each HTTP call made from inside these scripts or libraries that are using python-requests.
Set method_whitelist=False to retry for POST API calls as well. E.g. If you are communicating with GraphQL endpoint all calls are POST. By default only ['HEAD', 'TRACE', 'GET', 'PUT', 'OPTIONS', 'DELETE'] are retried as they are deemed safe / idempotent operations.
It will wait until the response arrives before the rest of your program will execute. If you want to be able to do other things, you will probably want to look at the asyncio or multiprocessing modules. Chad S. Chad S.
What can Requests do? Requests will allow you to send HTTP/1.1 requests using Python. With it, you can add content like headers, form data, multipart files, and parameters via simple Python libraries. It also allows you to access the response data of Python in the same way.
This snippet of code will make all HTTP requests from the same session retry for a total of 5 times, sleeping between retries with an increasing backoff of 0s, 2s, 4s, 8s, 16s (the first retry is done immediately). It will retry on basic connectivity issues (including DNS lookup failures), and HTTP status codes of 502, 503 and 504.
import logging import requests from requests.adapters import HTTPAdapter, Retry logging.basicConfig(level=logging.DEBUG) s = requests.Session() retries = Retry(total=5, backoff_factor=1, status_forcelist=[ 502, 503, 504 ]) s.mount('http://', HTTPAdapter(max_retries=retries)) s.get("http://httpstat.us/503")
See Retry class for details.
This is a snippet of code I used to retry for the petitions made with urllib2. Maybe you could use it for your purposes:
retries = 1 success = False while not success: try: response = urllib2.urlopen(request) success = True except Exception as e: wait = retries * 30; print 'Error! Waiting %s secs and re-trying...' % wait sys.stdout.flush() time.sleep(wait) retries += 1
The waiting time grows incrementally to avoid be banned from server.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With