Annotating/Fetching/mapping Testrail test case ID to the local Java TestNG test (Jenkins Test Rail Integration)

Question

I am trying to use Testrail as a test case management system and so, integrating testrail with the Jenkins would be useful.

This is what I want to achieve:

Lets say I manually create three test cases in testrail with case ID's C1, C2 and C3 and these test cases will have some unique automated test names such as A1, A2, and A3 (in more info, there will be a field in testrail with such a unique information)

When I hit "Start Automated Tests" button and run a Jenkins job from testrail (considering I have already implemented UI script for testrail that has this button):

, I want to run a script/something that takes the case ID's of the selected test cases and annotate those IDs to the actual Java tests temporarily so that it can run those specific tests and post results back to the Testrail.

Approach I can think of:

When I hit "Start Automated Tests" button on Testrail, I can make a script to run to create an XML file that will include the desired selected test cases from Testrail. This XML will then be provided as a default input to the Jenkins job and it will run the test cases mentioned in the XML file. This XML will be temporary and will be replaced everytime the selection is made from the Testrail. However, how do you do it? I am a newbie to the Testrail and read its API and looks like API will be useful to post the results back to the Testrail. But, how do we achieve the mapping of the ID's?

Also, any advise on posting results back to the Testrail will be useful.

Answer 1

This isn't TestNG specific, but you can make a custom annotation in java. You can update a TestRail test in a test run through the api either by the test ID (using add_result), or both the case id and run id(using add_result_for_case). http://docs.gurock.com/testrail-api2/reference-results

The case id doesn't ever change, so you can just hard code these in your tests.

Here is what I'm using for this purpose:

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface TestData
{
    int testId() default 0;
    String[] tags() default "";
}

My test method then looks like this(Using Junit, but shouldn't be much different with other frameworks):

@Test
@TestData(
        testId = 177,
        tags = {"smoke", "authentication"}
)
public void testName()
{
    //Do the test
}

I then use a JUnit specific way to get the test method name to use in my teardown method, but I'm sure there is a variety of ways to do that. Once you have the test method name here is how I read the annotation:

@After
public void baseTearDown() throws Exception
{
    //Good place to record test results
    Method testMethod = getClass().getMethod(testName);
    if(testMethod.isAnnotationPresent(TestData.class))
    {
        TestData testData = testMethod.getAnnotation(TestData.class);
        //Do something with testData.testId();
        System.out.println("Test ID = " + testData.testId());
    }

    //other cleanups
}

This mkyong link gives some pretty basic examples of both creating an annotation and reading it with reflection. This is what I used to get started: https://www.mkyong.com/java/java-custom-annotations-example/

If you're starting the test run in your code, then you can just keep track of the test run id and use it as needed. If not, my preference is to define and set some environment variables using Jenkins or other scripts that your code can read from so you don't have to deal with passing around files for some really basic key value pairs