Python pytest pytest_exception_interact customize exception information from VCR.py exception

Question

Context

I have started using pytest-vcr which is a pytest plugin wrapping VCR.py which I have documented in this blog post on Advanced Python Testing.

It records all HTTP traffic to cassettes/*.yml files on the first test run to save snapshots. Similar to Jest snapshot testing for web components.

On subsequent test runs, if a request is malformed, it won't find a match and throws an exception saying that recording new requests is forbidden and it did not find an existing recording.

Question

VCR.py raises a CannotOverwriteExistingCassetteException which is not particularly informative as to why it didn't match.

How do I leverage pytest pytest_exception_interact hooks to replace this exception with a more informative one leveraging fixture information?

I dove into my site-packages where VCR.py is pip installed and rewrote how I want it to handle the exception. I just need to know how to get this pytest_exception_interact hook to work correctly to access the fixtures from that test node (before it gets cleaned up) and raise a different exception.

Example

Lets get the dependencies.

$ pip install pytest pytest-vcr requests

test_example.py:

import pytest
import requests

@pytest.mark.vcr
def test_example():
    r = requests.get("https://www.stackoverflow.com")
    assert r.status_code == 200

$ pytest test_example.py --vcr-record=once
...
test_example.py::test_example PASSED 
...
$ ls cassettes/
cassettes/test_example.yml
$ head cassettes/test_example.yml
interactions:
- request:
    uri: https://wwwstackoverflow.com
    body: null
    headers:
        Accept:
        - '*/*'
$ pytest test_example.py --vcr-record=none
...
test_example.py::test_example PASSED 
...

Now change the URI in the test to "https://www.google.com":

test_example.py:

import pytest
import requests

@pytest.mark.vcr
def test_example():
    r = requests.get("https://www.google.com")
    assert r.status_code == 200

And run the test again to detect the regression:

$ pytest test_example.py --vcr-record=none
E               vcr.errors.CannotOverwriteExistingCassetteException: No match for the request (<Request (GET) https://www.google.com/>)
...

I can add a conftest.py file to the root of my test structure to create a local plugin, and I can verify that I can intercept the exception and inject my own using:

conftest.py

import pytest
from vcr.errors import CannotOverwriteExistingCassetteException
from vcr.config import VCR
from vcr.cassette import Cassette

class RequestNotFoundCassetteException(CannotOverwriteExistingCassetteException):
    ...

@pytest.fixture(autouse=True)
def _vcr_marker(request):
    marker = request.node.get_closest_marker("vcr")
    if marker:
        cassette = request.getfixturevalue("vcr_cassette")
        vcr = request.getfixturevalue("vcr")
        request.node.__vcr_fixtures = dict(vcr_cassette=cassette, vcr=vcr)
    yield


@pytest.hookimpl(hookwrapper=True)
def pytest_exception_interact(node, call, report):
    excinfo = call.excinfo
    if report.when == "call" and isinstance(excinfo.value, CannotOverwriteExistingCassetteException):
        # Safely check for fixture pass through on this node
        cassette = None
        vcr = None
        if hasattr(node, "__vcr_fixtures"):
            for fixture_name, fx in node.__vcr_fixtures.items():
                vcr = fx if isinstance(fx, VCR)
                cassette = fx if isinstance(fx, Cassette)

        # If we have the extra fixture context available...
        if cassette and vcr:
            match_properties = [f.__name__ for f in cassette._match_on]
            cassette_reqs = cassette.requests
            #  filtered_req = cassette.filter_request(vcr._vcr_request)
            #  this_req, req_str = __format_near_match(filtered_req, cassette_reqs, match_properties)

            # Raise and catch a new excpetion FROM existing one to keep the traceback
            # https://stackoverflow.com/a/24752607/622276
            # https://docs.python.org/3/library/exceptions.html#built-in-exceptions
            try:
                raise RequestNotFoundCassetteException(
                    f"\nMatching Properties: {match_properties}\n" f"Cassette Requests: {cassette_reqs}\n"
                ) from excinfo.value
            except RequestNotFoundCassetteException as e:
                excinfo._excinfo = (type(e), e)
                report.longrepr = node.repr_failure(excinfo)

This is the part where the documentation on the internet gets pretty thin.

How do I access the vcr_cassette fixture and return a different exception?

What I want to do is get the filtered_request that was attempting to be requested and the list of cassette_requests and using the Python difflib standard library produce deltas against the information that diverged.

PyTest Code Spelunking

The internals of running a single test with pytest triggers pytest_runtest_protocol which effectively runs the following three call_and_report calls to get a collection of reports.

src/_pytest/runner.py:L77-L94

def runtestprotocol(item, log=True, nextitem=None):
    # Abbreviated
    reports = []
    reports.append(call_and_report(item, "setup", log))
    reports.append(call_and_report(item, "call", log))
    reports.append(call_and_report(item, "teardown", log))
    return reports

So I'm after modifying the report at the call stage... but still no clue how I get access to the fixture information.

src/_pytest/runner.py:L166-L174

def call_and_report(item, when, log=True, **kwds):
    call = call_runtest_hook(item, when, **kwds)
    hook = item.ihook
    report = hook.pytest_runtest_makereport(item=item, call=call)
    if log:
        hook.pytest_runtest_logreport(report=report)
    if check_interactive_exception(call, report):
        hook.pytest_exception_interact(node=item, call=call, report=report)
    return report

It looks like there are some helper methods for generating a new ExceptionRepresentation so I updated the conftest.py example.

src/_pytest/reports.py:L361

longrepr = item.repr_failure(excinfo)

UPDATE #1 2019-06-26: Thanks to some pointers from @hoefling in the comments I updated my conftest.py.

• Correctly re-raising the exception using the raise ... from ... form.
• Override the _vcr_marker to attach the vcr and vcr_cassette fixtures to the request.node which represent that individual test item.
• Remaining: Get access to the intercepted request from the patched VCRConnection...

UPDATE #2 2019-06-26

It would seem impossible to get at the VCRHTTPConnections that were patched in creating the cassette context manager. I have opened up the following pull request to pass as arguments when the exception is thrown, to then catch and handle arbitrarily down stream.

https://github.com/kevin1024/vcrpy/pull/445

Related

Related questions that are informative but still don't answer this question.

• Customizing error message for specific exceptions in pytest

Answer 1

Thanks to comments and guidance in the comments from @hoefling.

I could attach the cassette fixture to the request.node in a conftest.py local plugin overriding the pytest-vcr marker...

@pytest.fixture(autouse=True)
def _vcr_marker(request):
    marker = request.node.get_closest_marker("vcr")
    if marker:
        cassette = request.getfixturevalue("vcr_cassette")
        vcr = request.getfixturevalue("vcr")
        request.node.__vcr_fixtures = dict(vcr_cassette=cassette, vcr=vcr)
    yield

But I needed more than the cassette to get to my solution.

Ingredients

• Use the pytest_exception_interact hook
• Pull request to VCR.py https://github.com/kevin1024/vcrpy/pull/446
  • PR #439 and PR #441 by arthurHamon2 has been huge to fix the test suite and also integrate matcher differencing outputs.
• BONUS: Use the raise ... from ... form of throwing an exception

Recipe

Get latest VCRpy

These patches were released in vcrpy v2.1.0

pip install vcrpy==2.1.0

Override the pytest_exception_interact hook

In the root of your test directory create a conftest.py to create a local plugin that overrides the pytest_exception_interact hook.

@pytest.hookimpl(hookwrapper=True)
def pytest_exception_interact(node, call, report):
    """Intercept specific exceptions from tests."""
    if report.when == "call" and isinstance(call.excinfo.value, CannotOverwriteExistingCassetteException):
        __handle_cassette_exception(node, call, report)

    yield

Extract the Cassette and the Request from the exception.

# Define new exception to throw
class RequestNotFoundCassetteException(Exception):
   ...

def __handle_cassette_exception(node, call, report):
    # Safely check for attributes attached to exception
    vcr_request = None
    cassette = None
    if hasattr(call.excinfo.value, "cassette"):
        cassette = call.excinfo.value.cassette
    if hasattr(call.excinfo.value, "failed_request"):
        vcr_request = call.excinfo.value.failed_request

    # If we have the extra context available...
    if cassette and vcr_request:

        match_properties = [f.__name__ for f in cassette._match_on]
        this_req, req_str = __format_near_match(cassette.requests, vcr_request, match_properties)

        try:
            raise RequestNotFoundCassetteException(f"{this_req}\n\n{req_str}\n") from call.excinfo.value
        except RequestNotFoundCassetteException as e:
            call.excinfo._excinfo = (type(e), e)
            report.longrepr = node.repr_failure(call.excinfo)