Related to older version of requests question: Can I set max_retries for requests.request?
I have not seen an example to cleanly incorporate max_retries
in a requests.get()
or requests.post()
call.
Would love a
requests.get(url, max_retries=num_max_retries))
implementation
To solve the requests "ConnectionError: Max retries exceeded with url", use a Retry object and specify how many connection-related errors to retry on and set a backoff factor to apply between attempts.
The POST method is used to send data mostly through a form to the server for creating or updating data in the server. The requests module provides us with post method which can directly send the data by taking the URL and value of the data parameter.
A quick search of the python-requests docs will reveal exactly how to set max_retries
when using a Session
.
To pull the code directly from the documentation:
import requests
s = requests.Session()
a = requests.adapters.HTTPAdapter(max_retries=3)
b = requests.adapters.HTTPAdapter(max_retries=3)
s.mount('http://', a)
s.mount('https://', b)
s.get(url)
What you're looking for, however, is not configurable for several reasons:
Requests no longer provides a means for configuration
The number of retries is specific to the adapter being used, not to the session or the particular request.
If one request needs one particular maximum number of requests, that should be sufficient for a different request.
This change was introduced in requests 1.0 over a year ago. We kept it for 2.0 purposefully because it makes the most sense. We also will not be introducing a parameter to configure the maximum number of retries or anything else, in case you were thinking of asking.
Edit Using a similar method you can achieve a much finer control over how retries work. You can read this to get a good feel for it. In short, you'll need to import the Retry
class from urllib3
(see below) and tell it how to behave. We pass that on to urllib3
and you will have a better set of options to deal with retries.
from requests.packages.urllib3 import Retry
import requests
# Create a session
s = requests.Session()
# Define your retries for http and https urls
http_retries = Retry(...)
https_retries = Retry(...)
# Create adapters with the retry logic for each
http = requests.adapters.HTTPAdapter(max_retries=http_retries)
https = requests.adapters.HTTPAdapter(max_retries=https_retries)
# Replace the session's original adapters
s.mount('http://', http)
s.mount('https://', https)
# Start using the session
s.get(url)
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With