Get the number of passed and failed tests with their name from command line

Question

I am writing Python script which will run mvn test on different folders, and I want to get number of passed test and failed tests and their names from script.

Now I just have managed to run process and get it output

proc = subprocess.run(["mvn", "test", "-f", PROJ_PATH])
print(proc.stdout)

output:

   Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.715 s
[INFO] Finished at: 2016-10-12T18:59:11+03:00
[INFO] Final Memory: 10M/212M
[INFO] ------------------------------------------------------------------------

I know that I can use regexp and try to parse output, but may be there are some more appropriate ways to incorporate with Maven from Python or Bash.

Answer 1

There are multiple solutions, but no direct one... they would all involve some parsing (XML or text file).

XML reports

This is probably the safest and simple route. Surefire generates by default XML reports inside target/surefire-reports. There is one XML file per test class, and this file contains the results of the execution of the tests in that class. This XML follows a pre-defined XSD and guarantees a stable output.

Each XML file (for each test class) inside target/surefire-reports is named TEST-${testClass}.xml where ${testClass} is replaced with the fully qualified name of the test class. Its relevant content for a test class of my.test.MyTest would be:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="my.test.MyTest" tests="4" errors="1" skipped="1" failures="1">
  <properties> <!-- omitted, contains system properties for the test --> </properties>
  <testcase name="test" classname="my.test.MyTest" time="0.096">
    <failure></failure>
  </testcase>
  <testcase name="test2" classname="my.test.MyTest" time="0.001">
    <error></error>
  </testcase>
  <testcase name="test3" classname="my.test.MyTest" time="0.002"/>
  <testcase name="test4" classname="my.test.MyTest" time="0">
    <skipped/>
  </testcase>
</testsuite>

(There are other attributes but they are not relevant here). Basically, the <testsuite> says there were 4 tests, which resulted in an error for 1, a failure for 1 and a skip for 1; so the remaining 1 was a success. More precisely, each <testcase> represent a test method through the name attribute, and the element inside represent its result. This can be parsed quite simply with regard to the 4 possible outcome for a test:

• failure (the assertions made in it were not verified): there is a <failure> element inside <testcase>.
• error (an exception was thrown, and it wasn't expected): there is a <error> element inside <testcase>.
• skip: there is a <skipped> element inside <testcase>.
• success: there is no element inside <testcase>.

If you want the fully qualified name of the test method, append the classname attribute (which is the qualified name of the test class) to the name attribute (which is the name of the test method).

Logs

If you configure the Surefire Plugin with the plain reportFormat with

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.19.1</version>
  <configuration>
    <reportFormat>plain</reportFormat>
  </configuration>
</plugin>

the logs will contain all the info wanted:

Running my.test.MyTest
Tests run: 4, Failures: 1, Errors: 1, Skipped: 1, Time elapsed: 0.169 sec <<< FAILURE! - in my.test.MyTest
test(my.test.MyTest)  Time elapsed: 0.112 sec  <<< FAILURE!
java.lang.AssertionError
    at my.test.MyTest.test(my.test.MyTest.java:16)

test2(my.test.MyTest)  Time elapsed: 0.001 sec  <<< ERROR!
java.lang.IllegalArgumentException: Exception
    at my.test.MyTest.test2(my.test.MyTest.java:21)

test3(my.test.MyTest)  Time elapsed: 0.002 sec
test4(my.test.MyTest) skipped

You can then have lots of fun greping this file with a regular expression looking for (.*)\((.*)\)\s+(?|(skipped)|Time elapsed:.*<<< (.*)): method name of the test is in group 1, fully qualified class name in group 2 and group 3 contains the result; if group 3 is null, then it's a success.