Is there a way for pytest to check if a log entry was made at Error level or higher?

Question

Python 3.8.0, pytest 5.3.2, logging 0.5.1.2.

My code has an input loop, and to prevent the program crashing entirely, I catch any exceptions that get thrown, log them as critical, reset the program state, and keep going. That means that a test that causes such an exception won't outright fail, so long as the output is still what is expected. This might happen if the error was a side effect of the test code but didn't affect the main tested logic. I would still like to know that the test is exposing an error-causing bug however.

Most of the Googling I have done shows results on how to display logs within pytest, which I am doing, but I can't find out if there is a way to expose the logs within the test, such that I can fail any test with a log at Error or Critical level.

Edit: This is a minimal example of a failing attempt:

test.py:

import subject
import logging
import pytest

@pytest.fixture(autouse=True)
def no_log_errors(caplog):
    yield  # Run in teardown
    print(caplog.records)
    # caplog.set_level(logging.INFO)
    errors = [record for record in caplog.records if record.levelno >= logging.ERROR]
    assert not errors

def test_main():
    subject.main()
    # assert False

subject.py:

import logging
logger = logging.Logger('s')
def main():
    logger.critical("log critical")

Running python3 -m pytest test.py passes with no errors. Uncommenting the assert statement fails the test without errors, and prints [] to stdout, and log critical to stderr.

Edit 2:

I found why this fails. From the documentation on caplog:

The caplog.records attribute contains records from the current stage only, so inside the setup phase it contains only setup logs, same with the call and teardown phases

However, right underneath is what I should have found the first time:

To access logs from other stages, use the caplog.get_records(when) method. As an example, if you want to make sure that tests which use a certain fixture never log any warnings, you can inspect the records for the setup and call stages during teardown like so:

@pytest.fixture
def window(caplog):
    window = create_window()
    yield window
    for when in ("setup", "call"):
        messages = [
            x.message for x in caplog.get_records(when) if x.levelno == logging.WARNING
        ]
        if messages:
            pytest.fail(
                "warning messages encountered during testing: {}".format(messages)
            )

However this still doesn't make a difference, and print(caplog.get_records("call")) still returns []

Answer 1

You can build something like this using the caplog fixture

here's some sample code from the docs which does some assertions based on the levels:

def test_baz(caplog):
    func_under_test()
    for record in caplog.records:
        assert record.levelname != "CRITICAL"
    assert "wally" not in caplog.text

since the records are the standard logging record types, you can use whatever you need there

here's one way you might do this ~more automatically using an autouse fixture:

@pytest.fixture(autouse=True)
def no_logs_gte_error(caplog):
    yield
    errors = [record for record in caplog.get_records('call') if record.levelno >= logging.ERROR]
    assert not errors

(disclaimer: I'm a core dev on pytest)