Parameterized unit test suites

Question

I am trying to set up some parameterized test suites, unfortunately without any luck so far. I have two set of parameters, and I would like to run multiple test cases (they are in different classes) with all possible combinations. I tried to do it with JUnit4, but I am unable to set it up correctly. This would be my basic idea:

1. TestSuite1.class sets up one set of parameters, then it starts TestSuite2.class.
2. TestSuite2.class sets up the second set of parameters, then it starts the actual test(s) that will use both parameters.

Meanwhile it seems it is not possible to set up both Suite.class and Parameterized.class in the RunWith annotation at the same time (according to google, Parameterized extends Suite, I get usually "no runnable method found" message if I use.)

This is how my code looks like basically:

TestSuite1.class:

@RunWith(Parameterized.class)
@Parameterized.SuiteClasses({TestSuite2.class})
//I have tried with @RunWith(Suite.class) and
//@Suite.SuiteClasses({TestSuite2.class}) annotations also - all combinations
public class TestSuite1{

  public TestSuite1(int number) {
    Params.first = number;
  } 

  @Parameters
  public static Collection<Object[]> parameters(){
    Object[][] data = new Object[][] { { 1 }, { 2 }, { 3 }, { 4 } };
    return Arrays.asList(data);
  }
}

TestSuite2.class looks the same as TestSuite1.class, except that I have added TestCase1.class to the suite instead of TestSuite2, and that it sets another variable in Params.

TestCase1.class:

public class TestCase1 {    
  @Test
  public void test1(){
    System.out.println("first: "+Params.first+" second: "+Params.second);
    Assert.assertTrue(true);
  }
}

I am open to all ideas - even with TestNG for example. I have tried it also (although today was the first time I saw it), but as I noticed the suites are a bit different than in JUnit. I would prefer not to set up XML files before testing, I would like to solve all set up programmatically.

Is what I am trying to achieve possible with any framework?

---

Update: With TestNG I have the following code:

Start.class:

public class Start {

public static void main(String[] args){
    TestListenerAdapter tla = new TestListenerAdapter();
    TestNG testng = new TestNG();
    testng.setTestClasses(new Class[] { FirstTest.class, SecondTest.class });
    testng.addListener(tla);
    testng.run();
}
}

Params.class:

public class Params {
@DataProvider(name = "param")
public static Object[][] createData() {
    Object[][] data = new Object[][] { { 1 }, { 2}, { 3}, { 4} };
    return data;
  }
}

FirstTest.class:

public class FirstTest {

@Test(dataProvider = "param", dataProviderClass = Params.class)
public static void printIt(int number){
    System.out.println("FirstTest: "+number);
}

}

SecondTest.class is the same as FirstTest.class. If I run this, it runs FirstTest 4 times, then it runs SecondTest 4 times. I would like to run FirstTest one time, and SecondTest one time also with the first set of parameters. Then I would like to run FirstTest and SecondTest one time, with the second set of parameters, etc.

I have tried to set setPreserveOrder(true), and tried all setParallel options also. On this way however the results are in kind of random order.

(It would be some selenium test. I am aware that tests should not depend on each other, but still it would be my desired way for this)

Answer 1

Although Parameterized extends Suite, it behaves totally different - in disrespect of the Liskov substitution principle. This is because normally the constructor Suite(Class<?>, RunnerBuilder) processes the @SuiteClasses annotation. But Parameterized(Class<?>) replaces this behaviour with a processing of @Parameters.

If you want to combine the behaviour of Suite and Parameterized you have to look outside of JUnit 4. E.g. you could implement your own Runner like Adam Hawkes already mentioned in another post here.

I did the same by myself and cobbled a library together that provides you with a ParameterizedSuite Runner: https://github.com/PeterWippermann/parameterized-suite

A parameterized test suite looks like this:

@RunWith(ParameterizedSuite.class)
@SuiteClasses({OneTest.class, TwoTest.class})
public class MyParameterizedTestSuite {
    @Parameters(name = "Parameters are {0} and {1}")
    public static Object[] params() {
        return new Object[][] {{'A',1}, {'B',2}, {'C',3}};
    }