TestNG retrying failed tests doesn't output the correct test results

Question

Setup: I have a class that extends the IRetryAnalyzer and have implemented a simple retry logic overriding the following method: public boolean retry(ITestResult result) {

I have another class that extends the class TestListenerAdapter that retries tests that failed until they pass or report failures. I have implemented my logic overriding the following method: public void onTestFailure(ITestResult result) {

Scenario: I have a total of 10 tests. 1 out of the 10 tests fail 2 times and pass on the 3rd attempt with my retry logic. The test results show the following: Total tests: 12, Failed: 2, Skipped: 0

What i would like is to output the correct number of tests run. And also disregard the 2 failures since the test passed at the end. So the result should look something like this: Total tests: 10, Failed:0, Skipped: 0

What am i missing here? Do i need to modify the ITestResult object? If yes, how?

FYI: I was able to achieve this using JUnit (implementing the TestRule interface).

Thanks in advance.

Answer 1

Please consider the following test results with max. 2 Retries:

1. Passed => Overall ok!
2. Failed, Passed => Overall ok!
3. Failed, Failed, Passed => Overall ok!
4. Failed, Failed, Failed => Overall Fail!

What i did is to create a TestNg listener which extends the default behaviour and does some magic after all tests are finished.

public class FixRetryListener extends TestListenerAdapter {

    @Override
    public void onFinish(ITestContext testContext) {
        super.onFinish(testContext);

        // List of test results which we will delete later
        List<ITestResult> testsToBeRemoved = new ArrayList<>();

        // collect all id's from passed test
        Set <Integer> passedTestIds = new HashSet<>();
        for (ITestResult passedTest : testContext.getPassedTests().getAllResults()) {
            passedTestIds.add(TestUtil.getId(passedTest));
        }

        Set <Integer> failedTestIds = new HashSet<>();
        for (ITestResult failedTest : testContext.getFailedTests().getAllResults()) {

            // id = class + method + dataprovider
            int failedTestId = TestUtil.getId(failedTest);

            // if we saw this test as a failed test before we mark as to be deleted
            // or delete this failed test if there is at least one passed version
            if (failedTestIds.contains(failedTestId) || passedTestIds.contains(failedTestId)) {
                testsToBeRemoved.add(failedTest);
            } else {
                failedTestIds.add(failedTestId);
            }
        }

        // finally delete all tests that are marked
        for (Iterator<ITestResult> iterator = testContext.getFailedTests().getAllResults().iterator(); iterator.hasNext(); ) {
            ITestResult testResult = iterator.next();
            if (testsToBeRemoved.contains(testResult)) {
                iterator.remove();
            }
        }

    }

}

Basically i do 2 things:

1. Collect all passed test. If i encounter a failed test with at least one passed test i remove the failed test (That would cover case 2 and 3 from above)
2. Iterate over all failed test. If i encounter a failed test which previously failed i remove the current failed result. (That would cover case 3 and 4 actually). That also means i will only keep the first failed result if there are several failed results.

To identify a testresult i use the following simple hash function:

public class TestUtil {

    public static int getId(ITestResult result) {
        int id = result.getTestClass().getName().hashCode();
        id = 31 * id + result.getMethod().getMethodName().hashCode();
        id = 31 * id + (result.getParameters() != null ? Arrays.hashCode(result.getParameters()) : 0);
        return id;
    }
}

This approach does work with dataproviders as well but has one small limitation! If you use dataproviders with random values you'll run into problems:

@DataProvider(name = "dataprovider")
public Object[][] getData() {
    return new Object[][]{{System.currentTimeMillis()}};
}

For failed tests the dataprovider is reevaluated. Therefore you will get a new timestamp and the result1 and result2 for the same mehod will result in different hash values. Solution would be to include the parameterIndex in the getId() Method instead of parameters but it seems such a value is not included in ITestResult.

See this simple example as proof of concept:

@Listeners(value = FixRetryListener.class)
public class SimpleTest {

    private int count = 0;

    @DataProvider(name = "dataprovider")
    public Object[][] getData() {
        return new Object[][]{{"Run1"},{"Run2"}};
    }

    @Test(retryAnalyzer = RetryAnalyzer.class, dataProvider = "dataprovider")
    public void teste(String testName) {
        count++;
        System.out.println("---------------------------------------");
        System.out.println(testName + " " + count);
        if (count % 3 != 0) {
            Assert.fail();
        }

        count = 0;
    }

}

Will yield in:

Total tests run: 2, Failures: 0, Skips: 0