Custom test method name in TestNG reports

Question

I am working on a project where I need to invoke TestNG programatically(using data providers). Things are fine except that in the report, we are getting the name of the @Test method, which is a generic one to handle many cases. What we would like is to get a meaningful name in the report.

I was researching on this and found 3 ways, but unfortunately, all are failing for me.

1) Implement ITest

I have found about this here and here

I am setting the name I want as soon as I enter the @Test method(For all 3 ways i tried,this is how I am setting the name).This name is returned from getTestName(). What i observed is that getTestName() is getting called before and after my @Test. Initially, it is returning null(for handling NullPointerException, I return "" instead of null) and later it returns correct value. But i dont see this getting reflected in the report

Edit:Also tried setting the name from@BeforeMethod as suggested by artdanil

2 and 3

Both are based on solutions given in the second link above

By overriding setName in XmlSuite, I am getting

Exception in thread "main" java.lang.AssertionError: l should not be null
        at org.testng.ClassMethodMap.removeAndCheckIfLast(ClassMethodMap.java:58)
        at org.testng.internal.TestMethodWorker.invokeAfterClassMethods(TestMethodWorker.java:208)
        at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:114)
        at org.testng.TestRunner.privateRun(TestRunner.java:767)
        ...

By overriding toString(), I see these in logs (with my comments) but no updates in report

[2013-03-05 14:53:22,174] (Main.java:30) - calling execute 
    [2013-03-05 14:53:22,346] GenericFunctionTest.<init>(GenericFunctionTest.java:52) - inside constructor
    [2013-03-05 14:53:22,372] GenericFunctionTest.toString(GenericFunctionTest.java:276) - returning **//this followed by 3 invocations before arriving at @Test method**
    [2013-03-05 14:53:22,410] GenericFunctionTest.toString(GenericFunctionTest.java:276) - returning 
    [2013-03-05 14:53:22,416] GenericFunctionTest.toString(GenericFunctionTest.java:276) - returning 
    [2013-03-05 14:53:22,455] GenericFunctionTest.toString(GenericFunctionTest.java:276) - returning 
    [2013-03-05 14:53:22,892] GenericFunctionTest.<init>(GenericFunctionTest.java:52) - inside constructor 
    [2013-03-05 14:53:23,178] GenericFunctionTest.toString(GenericFunctionTest.java:276) - returning **//again blank as i havent set it yet**
    [2013-03-05 14:53:23,182] GenericFunctionTest.getResult(GenericFunctionTest.java:69) - inside with test case:TestCase{signature=Signature{...}}**//I am setting it immedietely after this**
    [2013-03-05 14:53:23,293] GenericFunctionTest.toString(GenericFunctionTest.java:276) - returning MyMethodName **//What i want**
    [2013-03-05 14:53:23,299] GenericFunctionTest.toString(GenericFunctionTest.java:276) - returning MyMethodName **// again**

Edit: tried again all 3 by hardcoding a value rather than setting it on entry of my test method. But same results

Answer 1

I had the same problem, and found that it helps to set the field storing test case name in the method annotated with @BeforeMethod, using native injection of TestNG to provide method name and test parameters. The test name is taken from test parameters supplied by the DataProvider. If your test method does not have parameters, just report the method name.

//oversimplified for demontration purposes
public class TestParameters {
    private String testName = null;
    private String testDescription = null;

    public TestParameters(String name,
                          String description) {
        this.testName = name;
        this.testDescription = description;
    }

    public String getTestName() {
        return testName;
    }
    public String getTestDescription() {
        return testDescription;
    }
}

public class SampleTest implements ITest {
    // Has to be set to prevent NullPointerException from reporters
    protected String mTestCaseName = "";

    @DataProvider(name="BasicDataProvider")
    public Object[][] getTestData() {
        Object[][] data = new Object[][] {
                { new TestParameters("TestCase1", "Sample test 1")},
                { new TestParameters("TestCase2", "Sample test 2")},
                { new TestParameters("TestCase3", "Sample test 3")},
                { new TestParameters("TestCase4", "Sample test 4")},
                { new TestParameters("TestCase5", "Sample test 5") }
        };
        return data;
    }

    @BeforeMethod(alwaysRun = true)
    public void testData(Method method, Object[] testData) {
        String testCase = "";
        if (testData != null && testData.length > 0) {
            TestParameters testParams = null;
            //Check if test method has actually received required parameters
            for (Object testParameter : testData) {
                if (testParameter instanceof TestParameters) {
                    testParams = (TestParameters)testParameter;
                    break;
                }
            }
            if (testParams != null) {
                testCase = testParams.getTestName();
            }
        }
        this.mTestCaseName = String.format("%s(%s)", method.getName(), testCase);
    }

    @Override
    public String getTestName() {
        return this.mTestCaseName;
    }

    @Test(dataProvider="BasicDataProvider")
    public void testSample1(TestParameters testParams){
        //test code here
    }

    @Test(dataProvider="BasicDataProvider")
    public void testSample2(TestParameters testParams){
        //test code here
    }

    @Test
    public void testSample3(){
        //test code here
    }
}

EDIT: Based on the comments below, I realized a sample from report will be useful.

Extract from the report from running code above:

<testng-results skipped="0" failed="0" total="5" passed="5">
  <suite name="SampleTests" duration-ms="2818" started-at="<some-time>" finished-at="<some-time>">
    <test name="Test1" duration-ms="2818" started-at="<some-time>" finished-at="<some-time>">
        <test-method 
            status="PASS" 
            signature="testSample1(org.example.test.TestParameters)[pri:0, instance:org.example.test.TimeTest@c9d92c]"
            test-instance-name="testSample1(TestCase5)"
            name="testSample1" 
            duration-ms="1014"
            started-at="<some-time-before>" 
            data-provider="BasicDataProvider" 
            finished-at="<some-time-later>" >
            <!-- excluded for demonstration purposes -->
        </test-method>
        <!-- the rest of test results excluded for brevity -->
    </test>
  </suite>
</testng-result>

Note, that the value returned from getTestName() method is in the test-instance-name attribute, and not in the name attribute.

Answer 2

I ran into a similar problem. First I implemented the ITest strategy already mentioned. And this is part of the solution, but not completely.

TestNG, for some reason, when building different reports, calls getName() on the test while building the report. This is fine if you are not using a data provider to generate different runs and set a unique name for each run by using the ITest strategy. If you are using a data provider to generate multiple runs of the same test and want each run to have a unique name then there is a problem. As the ITest strategy leaves the name for the test as the name set by the last run.

So I had to implement a very custom getName(). SOme assumptions (in my particular case):

1. Only three reports are run: TestHTMLReporter, EmailableReporter, XMLSuiteResultWriter.
2. When ever get name is not called as a result of one of the assumed reporters, then returning the currently set name is fine.
3. When a reporter is running, it makes its getName() calls in order and only 1 time for each run.

public String getTestName()
{
    String name = testName;
    StackTraceElement[] stackTrace = Thread.currentThread().getStackTrace();//.toString();
    if(calledFrom(stackTrace, "XMLSuiteResultWriter"))
    {
        name = testNames.size()>0?testNames.get(xmlNameIndex<testNames.size()?xmlNameIndex:0):"undefined";
        xmlNameIndex++;
        if(xmlNameIndex>=testNames.size())
            xmlNameIndex = 0;
    }
    else if(calledFrom(stackTrace, "EmailableReporter"))
    {
        name = testNames.size()>0?testNames.get(emailNameIndex<testNames.size()?emailNameIndex:0):"undefined";
        emailNameIndex++;
        if(emailNameIndex>=testNames.size())
            emailNameIndex = 0;
    }
    if(calledFrom(stackTrace, "TestHTMLReporter"))
    {
        if(testNames.size()<0)
        {
            name = "undefined";
        }
        else
        {
            if(htmlNameIndex < testNamesFailed.size())
            {
                name = testNamesFailed.get(htmlNameIndex);
            }
            else
            {
                int htmlPassedIndex = htmlNameIndex - testNamesFailed.size();
                if(htmlPassedIndex < testNamesPassed.size())
                {
                    name = testNamesPassed.get(htmlPassedIndex);
                }
                else
                {
                    name = "undefined";
                }
            }
        }
        htmlNameIndex++;
        if(htmlNameIndex>=testNames.size())
            htmlNameIndex = 0;
    }
    return name;
}

private boolean calledFrom(StackTraceElement[] stackTrace, String checkForMethod)
{
    boolean calledFrom = false;
    for(StackTraceElement element : stackTrace)
    {
        String stack = element.toString();
        if(stack.contains(checkForMethod))
            calledFrom = true;
    }
    return calledFrom;
}

When setting the name for the run (I did this in the @BeforeMethod(alwaysRun=true) method I defined following the ITest strategy) I added the name to an ArrayList testNames. But then the html report was not correct. Most of the other reports pulls the information in order, like the XMLSuiteResultWriter, but TestHTMLReporter gets the name by first getting all the names for failed tests and then the names for passing tests. So I had to implement to additional ArrayLists: testNamesFailed and testNamesPassed and add the test names to them when the test finished based on whether they passed or not.

And I will freely admit this is very much a hack and very fragile. Ideally, TestNG adds the tests to a collection while running, and gets the name from that collection instead of from the original test. If you have TestNG to run other reports you will have to figure out what order they request the names and what is a unique enough string to search for in the thread stack trace.

--Edit 1

Alternatively, use the ITest Strategy and the factory pattern (@factory annotations).

TestNG Using @Factory and @DataProvider

http://beust.com/weblog/2004/09/27/testngs-factory/

It does require some minor changes. This includes creating a constructor with the same parameters as the original test method. The test method now has no parameters. You can set the name in the new constructor and simply return that in the getTestName method. Make sure to remove the data provider specification from the test method.

Answer 3

If you want to change the name in the HTML report, it'll be a total hack. Here's how I did it:

public class NinjaTest {
...
...
@AfterMethod (alwaysRun = true)
public void afterMethod(ITestResult result, Method method) {
    try {
        //I have XML test suites organized in directories. 
        String xmlFile = result.getTestContext().getCurrentXmlTest().getSuite().getFileName();
        String suiteName = xmlFile.substring(xmlFile.lastIndexOf("\\") + 1, xmlFile.lastIndexOf(".xml"));
        String pathToFile = xmlFile.substring(0, xmlFile.lastIndexOf("\\") );
        String directory = pathToFile.substring(pathToFile.lastIndexOf("\\") + 1);
        String testMethodName = String.format("%s/%s - %s", directory, suiteName, method.getName());

        //Total hack to change display name in HTML report  ＼(^o^)／ 
        Field methodName = org.testng.internal.BaseTestMethod.class.getDeclaredField("m_methodName");
        methodName.setAccessible(true);
        methodName.set(result.getMethod(), testMethodName);
    } catch (Exception e) {
        // Eh....  ¯\_(ツ)_/¯
        e.printStackTrace();
    }
}
...
...