How to optimize testng and seleniums tests

Question

For my internship, I have to use TestNG and selenium for testing a web-application. But I have a problem, sometimes selenium or the Browser is not working for some random reason, so a working test is marked as "failed". To avoid that, I can use the annotation @Test(invocationCount = 4, successPercentage = 25), then if the test succeeds one time, the test is marked as "Succeed", that's good but the problem is that this solution multiply the time for testing by 4, this is not very efficient.

What I can do to decrease the time for testing, is to write some rule "if the test failed, rerun this test (and only if the test has failed), and if it worked the second, third, or the fourth time, then mark this test as "succeed" " So I can avoid these random bugs. But I've not found how to write this rule, I saw that we can add a listener, so we have a method called "onTestFailure" so I can do something when the test has failed but I don't know how to re-run the test.

I also found testng-failed.xml where all the failed tests are saved, so we can run this xml file for rerun these tests, but this will erase the report from the previous first run, but I want just that the failed tests are marked as "succeed" if the second run is successful. (I have integrated testNG/selenium to Jenkins, so I have a graph with all tests, so this method is not very adapted, but this method don't multiply the time for testing by 4 and this is what I want)

So if you have any clue for how to do that, it would be very nice.

Answer 1

You can use a combination of the IRetryAnalyzer, a Listener and a custom reporter to do what you are looking for.

IRetryAnalyzer:

public class RetryAnalyzer implements IRetryAnalyzer  { 
private int count = 0; 
// this number is actually twice the number
// of retry attempts will allow due to the retry
// method being called twice for each retry
private int maxCount = 6; 
protected Logger log;
private static Logger testbaseLog;

static {
    PropertyConfigurator.configure("test-config/log4j.properties");
    testbaseLog = Logger.getLogger("testbase.testng");
}

public RetryAnalyzer()
{
    testbaseLog.trace( " ModeledRetryAnalyzer constructor " + this.getClass().getName() );
    log = Logger.getLogger("transcript.test");
}

@Override 
public boolean retry(ITestResult result) { 
    testbaseLog.trace("running retry logic for  '" 
            + result.getName() 
            + "' on class " + this.getClass().getName() );
        if(count < maxCount) {                     
                count++;                                    
                return true; 
        } 
        return false; 
}
}

RetryListener:

public class RetryTestListener extends TestListenerAdapter  {
private int count = 0; 
private int maxCount = 3; 

@Override
public void onTestFailure(ITestResult result) {     
    Logger log = Logger.getLogger("transcript.test");
    Reporter.setCurrentTestResult(result);

    if(result.getMethod().getRetryAnalyzer().retry(result)) {    
        count++;
        result.setStatus(ITestResult.SKIP);
        log.warn("Error in " + result.getName() + " with status " 
                + result.getStatus()+ " Retrying " + count + " of 3 times");
        log.info("Setting test run attempt status to Skipped");                 
    } 
    else
    {
        count = 0;
        log.error("Retry limit exceeded for " + result.getName());
    }       

    Reporter.setCurrentTestResult(null);
}

@Override
public void onTestSuccess(ITestResult result)
{
    count = 0;
}

However, there appears to be a bug within TestNG that actually causes some of the test results to be reported as both skipped AND failed. To prevent this, I recommend that you override whatever Reporter you wish to use and include a method such as the one included below:

private IResultMap removeIncorrectlyFailedTests(ITestContext test)
{     
  List<ITestNGMethod> failsToRemove = new ArrayList<ITestNGMethod>();
  IResultMap returnValue = test.getFailedTests();

  for(ITestResult result : test.getFailedTests().getAllResults())
  {
    long failedResultTime = result.getEndMillis();          

    for(ITestResult resultToCheck : test.getSkippedTests().getAllResults())
    {
        if(failedResultTime == resultToCheck.getEndMillis())
        {
            failsToRemove.add(resultToCheck.getMethod());
            break;
        }
    }

    for(ITestResult resultToCheck : test.getPassedTests().getAllResults())
    {
        if(failedResultTime == resultToCheck.getEndMillis())
        {
            failsToRemove.add(resultToCheck.getMethod());
            break;
        }
    }           
  }

  for(ITestNGMethod method : failsToRemove)
  {
      returnValue.removeResult(method);
  }  

  return returnValue;
}

After all of this is done, you can add the reporter using using .addListener and specify the retryAnalyzer in the @Test annotation.

Answer 2

You need not implement onTestFailure. TestNG calls retry automatically when test fails. So no need to override onTestFailure. this causes retry method 2 calls. I implemented retry as below.

private final Map rerunCountForTesCase = new HashMap();
    @Override
    public boolean retry(ITestResult result) {
                // here i have unique test case IDs for each of test method. 
                // So using utility method to get it. You can use method calss+name combination instead of testcaseid like me
        String executingTestCaseID = ReportingUtilities.getTestCaseId(result);
        if(rerunCountForTesCase.containsKey(executingTestCaseID))
        {
            count = rerunCountForTesCase.get(executingTestCaseID);
        }
        else
        {
            rerunCountForTesCase.put(executingTestCaseID, 0);
        }
        if (count 0)
                {
                    logInfo(tcID,"Sleeping for "+timeInSecs+ " secs before rerunning the testcase");
                    Thread.sleep(timeInSecs * CommonFwBase.SHORTWAIT );
                }
            } catch (InterruptedException e) {
                logError(null, e.getMessage());

            }

            rerunCountForTesCase.put(executingTestCaseID, ++count);
            return true;
        }
        return false;

    }

In the above thread retry getting called twice because of implementation of onTestFailure. I remove the failed results so that when you retry It uses latest result. Otherwise if you have dependency test method it get skipped(though on retry it passed as it uses first result).You might have to handle failed retried results while reporting. You might have to remove the tests that are passing after retry like this.

        m_ptests = suiteTestContext.getPassedTests();
        m_ftests = suiteTestContext.getFailedTests();
        m_stests = suiteTestContext.getSkippedTests();

        List<ITestNGMethod> methodsToRemove = new ArrayList<ITestNGMethod>();

        for(ITestResult failed_result : m_ftests.getAllResults())
        {
            String failed_testName = failed_result.getMethod().getMethodName();
            String failingTest_className = failed_result.getClass().getName();
            for(ITestResult passed_result : m_ptests.getAllResults())
            {
                String passing_testName = passed_result.getMethod().getMethodName();
                String passingTest_className = failed_result.getClass().getName();
                if(failed_testName.equals(passing_testName) &&  
                        passingTest_className.equals(failingTest_className) ))
                {
                    if(passed_result.getEndMillis() > failed_result.getEndMillis())
                    {

                        methodsToRemove.add(failed_result.getMethod());
                        break;
                    }

                }
            }
        }

        // remove the test that passed on retry
        for(ITestNGMethod failedMethodToRemove : methodsToRemove)
        {
            m_ftests.removeResult(failedMethodToRemove);
        }

Hope it helps to understand retry better.