Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

History of retries using Request Library

I'm building a new retry feature in my Orchestrate script and I want to know how many times, and, if possible, what error my request method got when trying to connect to a specific URL.

For now, I need this for logging purposes, because I'm working on a messaging system and I may need this 'retry' information to understand when and why I'm facing any kind of problem in HTTP requests, once I work in a micro-service environment.

So far, I debugged and certify that retries are working as expected (I have a mocked flask server for all micro services that we use), but I couldn't find a way to got the 'retries history' data.

In other words, for example, I want to see if a specific micro-service may respond only after the third request, and those kind of thing.

Below is the code that I'm using now:

from requests import exceptions, Session
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

def open_request_session():
    # Default retries configs
    toggle = True #loaded from config file

    session = Session()

    if toggle:

        # loaded from config file as 
        parameters = {'total': 5, 
                      'backoff_factor': 0.0, 
                      'http_error_to_retry': [400]}well

        retries = Retry(total=parameters['total'],
                        backoff_factor=parameters['backoff_factor'],
                        status_forcelist=parameters['http_error_to_retry'],
                        # Do not force an exception when False
                        raise_on_status=False)

        session.mount('http://', HTTPAdapter(max_retries=retries))
        session.mount('https://', HTTPAdapter(max_retries=retries))

    return session

# Request
    with open_request_session() as request:
        my_response = request.get(url, timeout=10)

I see in the urllib3 documentation that Retry has a history attribute, but when I try to consult the attribute it is empty.

I don't know if I'm doing wrong or forgetting something, once Software Development is not my best skill.

So, I have two questions:

  1. Does anyone know a way to got this history information?
  2. How can I do create tests to verify if the retries behavior is working as expected? (So far I only test in debug mode)

I'm using Python 3.6.8.

I know that I can create a while statement to 'control' this, but I'm trying to avoid complexity. And this is why I'm here, I'm looking for an alternative based on Python and community best practices.

like image 726
DuanPsycho Avatar asked Sep 16 '25 09:09

DuanPsycho


1 Answers

A bit late, but I just figured this out myself so thought I'd share what I found.

Short answer:

response.raw.retries.history

will get you what you are looking for.

Long answer:

You cannot get the history off the original Retry instance created. Under the covers, urllib3 creates a new Retry instance for every attempt.

urllib3 does store the last Retry instance on the response when one is returned. However, the response from the requests library is a wrapper around the urllib3 response. Luckily, requests stores the original urllib3 response on the raw field.

like image 113
loesak Avatar answered Sep 17 '25 23:09

loesak



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!