Jenkins - How to find out which tests failed repeatedly?

Question

I am new to CI & Jenkins. I have a Java project which runs Testng based automated tests. The tests run regularly as a job in Jenkins. Sometimes, the job fails repeatedly for a long time. But, in each run, the number of test failures is different. I want to see which tests failed repeatedly in, say the last 5 runs. With that, I can reduce the number of test failures which I have to investigate. The failures which do not repeat all in the last 5 runs could be ignored or investigated later.

Can someone please suggest how I can find out which tests failed repeatedly in some consecutive runs ?

PS - Here is an example to explain why I want to see test failures in the past few runs.

Run 1 - test1 failed, test3 failed, test10 failed.
Run 2 - test3 failed, test17 failed.
Run 3 - test1 failed, test3 failed.

After looking at these 3 runs, we see that test3 fails in all the runs. But, the other tests pass in at least one of the three runs. I know that test3 used to pass before I started these 3 runs. This tells me that test3 might be failing due to a bug in the system, but the other tests might be failing due to intermittent issues in the system. This info allows me to decide which test to investigate first, i.e. test3. I can look at the other tests later if needed.

Answer 1

You can take the number of times a test failed from the build's object in a pipeline: you can make a small report and attach it to your build like so:

@NonCPS
def getRepeatedlyFailingTests(int timesFailedAtLeast = 2) {
  currentBuild.rawBuild.getAction(hudson.tasks.junit.TestResultAction.class)
    .getFailedTests()
    // Keep only tests that failed at least twice
    .findAll { it.age >= timesFailedAtLeast }
    .collect { [ "${it.className}.${it.name}".trim(), it.age ] }
}

def saveRepeatedlyFailingTestsReport() {
  def header = [ "test", "times-failed" ]
  def records = getRepeatedlyFailingTests()
  def report = "repeatedly-failing-tests.csv"
  writeCSV file: report, records: [ header ] + records, format: CSVFormat.EXCEL
  archiveArtifacts report
}

Call saveRepeatedlyFailingTestsReport after you gather your test results and you'll see the report in your latest build artifacts link.