Re-running complete class and not just @Test in TestNG

Question

I have been browsing stackoverflow for some days, trying to find how to re-run a whole test class, and not just an @Test step. Many say that this is not supported with TestNG and IRetryAnalyzer, whereas some have posted workarounds, that don't really work. Has anyone manage to do it? And just to clarify the reasons for this, in order to avoid answers that say that is not supported in purpose: TestNG is a tool not only for developers. Meaning that is also used from sw testers for e2e testing. E2e tests can have steps that depend each from the previous one. So yes it's valid to re-run whole test class, rather than simple @Test, which is easily can be done via IRetryAnalyzer.

An example of what I want to achieve would be:

public class DemoTest extends TestBase {

@Test(alwaysRun = true, description = "Do this")
public void testStep_1() {
    driver.navigate().to("http://www.stackoverflow.com");
    Assert.assertEquals(driver.getCurrentUrl().contains("stackoverflow)"));

}

@Test(alwaysRun = true, dependsOnMethods = "testStep_1", description = "Do that")
public void testStep_2() {
    driver.press("button");
    Assert.assertEquals(true, driver.elementIsVisible("button"));

}

@Test(alwaysRun = true, dependsOnMethods = "testStep_2", description = "Do something else")
public void testStep_3() {
   driver.press("button2");
Assert.assertEquals(true, driver.elementIsVisible("button"));

}

}

Let's say that testStep_2 fails, I want to rerun class DemoTest and not just testStep_2

Answer 1

Okay, I know that you probably want some easy property you can specify in your @BeforeClass or something like that, but we might need to wait for that to be implemented. At least I couldn't find it either.

The following is ugly as hell but I think it does the job, at least in a small scale, it is left to see how it behaves in more complex scenarios. Maybe with more time, this can be improved into something better.

Okay, so I created a Test Class similar to yours:

public class RetryTest extends TestConfig {

    public class RetryTest extends TestConfig {

        Assertion assertion = new Assertion();

        @Test(  enabled = true,
                groups = { "retryTest" },
                retryAnalyzer = TestRetry.class,
                ignoreMissingDependencies = false)
        public void testStep_1() {
        }

        @Test(  enabled = true,
                groups = { "retryTest" },
                retryAnalyzer = TestRetry.class,
                dependsOnMethods = "testStep_1",
                ignoreMissingDependencies = false)
        public void testStep_2() {
            if (fail) assertion.fail("This will fail the first time and not the second.");
        }

        @Test(  enabled = true,
                groups = { "retryTest" },
                retryAnalyzer = TestRetry.class,
                dependsOnMethods = "testStep_2",
                ignoreMissingDependencies = false)
        public void testStep_3() {
        }

        @Test(  enabled = true)
        public void testStep_4() {
            assertion.fail("This should leave a failure in the end.");
        }

    }

I have the Listener in the super class just in the case I'd like to extend this to other classes, but you can as well set the listener in your test class.

@Listeners(TestListener.class)
public class TestConfig {
   protected static boolean retrySuccessful = false;
   protected static boolean fail = true;
}

Three of the 4 methods above have a RetryAnalyzer. I left the testStep_4 without it to make sure that what I'm doing next doesn't mess with the rest of the execution. Said RetryAnalyzer won't actually retry (note that the method returns false), but it will do the following:

public class TestRetry implements IRetryAnalyzer {

    public static TestNG retryTestNG = null;

    @Override
    public boolean retry(ITestResult result) {
        Class[] classes = {CreateBookingTest.class};

        TestNG retryTestNG = new TestNG();
        retryTestNG.setDefaultTestName("RETRY TEST");
        retryTestNG.setTestClasses(classes);
        retryTestNG.setGroups("retryTest");
        retryTestNG.addListener(new RetryAnnotationTransformer());
        retryTestNG.addListener(new TestListenerRetry());
        retryTestNG.run();

        return false;
    }

}

This will create an execution inside of your execution. It won't mess with the report, and as soon as it finishes, it will continue with your main execution. But it will "retry" the methods within that group.

Yes, I know, I know. This means that you are going to execute your test suite forever in an eternal loop. That's why the RetryAnnotationTransformer. In it, we will remove the RetryAnalyzer from the second execution of those tests:

public class RetryAnnotationTransformer extends TestConfig implements IAnnotationTransformer {

    @SuppressWarnings("rawtypes")
    @Override
    public void transform(ITestAnnotation annotation, Class testClass, Constructor testConstructor, Method testMethod) {
        fail = false; // This is just for debugging. Will make testStep_2 pass in the second run.
        annotation.setRetryAnalyzer(null);
    }

}

Now we have the last of our problems. Our original test suite knows nothing about that "retry" execution there. This is where it gets really ugly. We need to tell our Reporter what just happened. And this is the part that I encourage you to improve. I'm lacking the time to do something nicer, but if I can, I will edit it at some point.

First, we need to know if the retryTestNG execution was successful. There's probably a million ways to do this better, but for now this works. I set up a listener just for the retrying execution. You can see it in TestRetry above, and it consists of the following:

public class TestListenerRetry extends TestConfig implements ITestListener {

    (...)

    @Override
    public void onFinish(ITestContext context) {
        if (context.getFailedTests().size()==0 && context.getSkippedTests().size()==0) {
            successful = true;
        }
    }

}

Now the Listener of the main suite, the one you saw above in the super class TestConfig will see if it the run happened and if it went well and will update the report:

public class TestListener extends TestConfig implements ITestListener , ISuiteListener {

    (...)

    @Override
    public void onFinish(ISuite suite) {

        if (TestRetry.retryTestNG != null) {

            for (ITestNGMethod iTestNGMethod : suite.getMethodsByGroups().get("retryTest")) {

                Collection<ISuiteResult> iSuiteResultList = suite.getResults().values();

                for (ISuiteResult iSuiteResult : iSuiteResultList) {

                    ITestContext iTestContext = iSuiteResult.getTestContext();
                    List<ITestResult> unsuccessfulMethods = new ArrayList<ITestResult>();

                    for (ITestResult iTestResult : iTestContext.getFailedTests().getAllResults()) {
                        if (iTestResult.getMethod().equals(iTestNGMethod)) {
                            iTestContext.getFailedTests().removeResult(iTestResult);
                            unsuccessfulMethods.add(iTestResult);
                        }
                    }

                    for (ITestResult iTestResult : iTestContext.getSkippedTests().getAllResults()) {
                        if (iTestResult.getMethod().equals(iTestNGMethod)) {
                            iTestContext.getSkippedTests().removeResult(iTestResult);
                            unsuccessfulMethods.add(iTestResult);
                        }
                    }

                    for (ITestResult iTestResult : unsuccessfulMethods) {
                        iTestResult.setStatus(1);
                        iTestContext.getPassedTests().addResult(iTestResult, iTestResult.getMethod());
                    }

                }

            }

        }


    }

}

The report should show now 3 tests passed (as they were retried) and one that failed because it wasn't part of the other 3 tests:

I know it's not what you are looking for, but I help it serves you until they add the functionality to TestNG.