Customizing pytest junitxml failure reports

Question

I am trying to introspect test failures and include additional data into the junit xml test report. Specifically, this is a suite of functional tests on an external product, and I want to include the product's logs into the failure reports.

Using the method found here, I was able to print the logs to stdout before executing the multicall which eventually show in jenkin's fail report. But I'm sure there's a better way to achieve this.

I tried to use the pytest_runtest_logreport hook to append the logs into the 'sections' attribute, which already contains the 'captured stdout' and 'captured stderr' streams. But the newly added sections do not make it to the xml file. I tried the above technique directly into the pytest_runtest_makereport hook as well, with similar results.

The release notes for pytest 2.7 states that using multicall support is being dropped for 2.8 and that @pytest.mark.hookwrapper is the new way to do it, however I can't seem to make that work at all - the "yield" returns None instead of a CallOutcome object (tried it in the makereport hook). And even if it returned something, I'm not sure I could add stuff to it that would show up in the xml report.

Is there any functionality I'm missing that will let me do this in a flexible way? (by flexible I mean: not being bound to stdout or logging calls like the capture-logs plugin)

Answer 1

EDIT: Since I needed access to the test item's funcargs (and test result) for my reporting, I was able to move the logic to the pytest_runtest_makereport(item, __multicall__) hook. The trick is to execute the multicall, which returns the report object:

@pytest.mark.tryfirst
def pytest_runtest_makereport(item, call, __multicall__):
    report = __multicall__.execute()
    # then I was able to manipulate report and get the same results as below

---

Bruno's answer gave me the motivation I needed to analyze this feature more thoroughly :)

So here's how it works:

def pytest_runtest_logreport(report):
    if report.failed:
        report.longrepr.sections.append(("Header", "Message", "-"))
        report.sections.append(("Captured stdout", "This is added to stdout"))
        report.sections.append(("Captured stderr", "This is added to stderr"))
        report.sections.append(("Custom Section", "This can only be seen in the console - the xml won't have it."))

The longrepr attribute is only available in case of failures. It takes a 3-tuple, the last value being a character used to decorate the decorate/surround the header. It will appear in the "failure" section of the report:

----------------------------------- Header ------------------------------------
Message

Custom sections will create additional result sections to be printed out to the console. But they won't make it to junitxml:

------------------------------- Custom Section --------------------------------
This can only be seen in the console - the xml won't have it.

The junitxml report only has 2 sections: out and err. To add custom text to it, you must create sections called "Captured std" and only those will make it to the xml file. Any other name will result in a custom section that will only be seen in the console.

Here's the resulting junitxml using the code above, with some reformatting for the sake of this post:

<?xml version="1.0" encoding="utf-8" ?> 
<testsuite errors="0" failures="1" name="pytest" skips="0" tests="1" time="0.646">
  <testcase classname="test_reporting" name="test_fail" time="0.000999927520752">
    <failure message="test failure">
      @ut def test_fail(): > assert 0, "It failed"
      E AssertionError: It failed 
      E assert 0 test_reporting.py:346: AssertionError
      ----------------------------------- Header ------------------------------------
      Message
    </failure> 
    <system-out>This is added to stdout</system-out> 
    <system-err>This is added to stderr</system-err> 
  </testcase>
</testsuite>

Answer 2

To add information to a test report (XML, console or otherwise) please take a look at reporting hooks, more specifically at pytest_runtest_logreport.