History of retries using Request Library

Question

I'm building a new retry feature in my Orchestrate script and I want to know how many times, and, if possible, what error my request method got when trying to connect to a specific URL.

For now, I need this for logging purposes, because I'm working on a messaging system and I may need this 'retry' information to understand when and why I'm facing any kind of problem in HTTP requests, once I work in a micro-service environment.

So far, I debugged and certify that retries are working as expected (I have a mocked flask server for all micro services that we use), but I couldn't find a way to got the 'retries history' data.

In other words, for example, I want to see if a specific micro-service may respond only after the third request, and those kind of thing.

Below is the code that I'm using now:

from requests import exceptions, Session
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

def open_request_session():
    # Default retries configs
    toggle = True #loaded from config file

    session = Session()

    if toggle:

        # loaded from config file as 
        parameters = {'total': 5, 
                      'backoff_factor': 0.0, 
                      'http_error_to_retry': [400]}well

        retries = Retry(total=parameters['total'],
                        backoff_factor=parameters['backoff_factor'],
                        status_forcelist=parameters['http_error_to_retry'],
                        # Do not force an exception when False
                        raise_on_status=False)

        session.mount('http://', HTTPAdapter(max_retries=retries))
        session.mount('https://', HTTPAdapter(max_retries=retries))

    return session

# Request
    with open_request_session() as request:
        my_response = request.get(url, timeout=10)

I see in the urllib3 documentation that Retry has a history attribute, but when I try to consult the attribute it is empty.

I don't know if I'm doing wrong or forgetting something, once Software Development is not my best skill.

So, I have two questions:

1. Does anyone know a way to got this history information?
2. How can I do create tests to verify if the retries behavior is working as expected? (So far I only test in debug mode)

I'm using Python 3.6.8.

I know that I can create a while statement to 'control' this, but I'm trying to avoid complexity. And this is why I'm here, I'm looking for an alternative based on Python and community best practices.

Answer 1

A bit late, but I just figured this out myself so thought I'd share what I found.

Short answer:

response.raw.retries.history

will get you what you are looking for.

Long answer:

You cannot get the history off the original Retry instance created. Under the covers, urllib3 creates a new Retry instance for every attempt.

urllib3 does store the last Retry instance on the response when one is returned. However, the response from the requests library is a wrapper around the urllib3 response. Luckily, requests stores the original urllib3 response on the raw field.