How to test all combinations of bad parameters?

Question

Let's assume a simple method which throws an IndexOutOfBoundsException for an invalid pair of indices (2d array).

How would I test, that the exception is thrown for all combinations of bad indices ?

(of course this test will not continue if one call throws an exception)

@Test(expected = Exception.class)
public void validateIndices(){
    check(0,-1);
    check(-1,0);
    check(0,COLS + 1);
    check(ROWS + 1, 0);
}

Is there a common way to test variations of parameters for a method ?

Answer 1

In addition to @Nicolas_Filotto answer, you can also use Junit Theories. It is more readable and unlike Parameterized it will execute the test with all possible combinations of parameters.

@RunWith(Theories.class)
public class MyTest {
    @DataPoints("cols")
    public static int[] rowValues(){
        return new int[]{0, -1, 1, 2};
    }
    @DataPoints("rows")
    public static int[] colValues(){
        return new int[]{0, -1, 4, 5};
    }

    @Theory
    public void upperBoundIsChecked(@FromDataPoints("cols") int col,
                                    @FromDataPoints("rows") int row){
        assumeTrue(row >= ROWS || col >= COLS);
        try {
            check(col, row);
            fail("Should have thrown IllegalArgumentException");
        } catch (IllegalArgumentException ignore){}
    }

    @Theory
    public void lowerBoundIsChecked(@FromDataPoints("cols") int col,
                                    @FromDataPoints("rows") int row){
        assumeTrue(row < 0 || col < 0);
        try {
            check(col, row);
            fail("Should have thrown IllegalArgumentException");
        } catch (IllegalArgumentException ignore){}
    }

    @Theory
    public void validIndicesNoException(@FromDataPoints("cols") int col,
                                        @FromDataPoints("rows") int row){
        assumeTrue(row >= 0 && col >= 0 && row < ROWS && col < COLS);
        try {
            check(col, row);
        } catch (Exception e){
            fail("Should not have thrown an exception: " + e.getMessage());
        }
    }

}

Each theory will check all possible combinations of rows and columns that match the assumptions of the theory.

Or, if your list of cols and rows are the same, it could be done even easier:

@RunWith(Theories.class)
public class MyTest {

   @DataPoints
   public static int[] values(){
      return new int[]{0, -1};
   }
   @Theory
   public void validateIndices(int col, int row){
      check(col,row);
   }
}

Answer 2

In your case, I would Parameterized my unit test to test all combinations with the same test more info here.

In your case it would look like this:

@RunWith(Parameterized.class)
public class MyTest {
    @Parameterized.Parameters
    public static Collection<Object[]> data() {
        return Arrays.asList(new Object[][] {
            { 0, -1 }, { -1, 0 }
        });
    }

    @Parameterized.Parameter
    public int row;

    @Parameterized.Parameter(value = 1)
    public int col;

    @Test(expected = IndexOutOfBoundsException.class)
    public void validateIndices(){
        check(row, col);
    }
}

Answer 3

The other answers are of course correct, in the sense of that they do what you are looking for, but they have one big conceptional downside: they can only check for error conditions ... that you have to manually specify.

Meaning: this kind of testing only proves that your "code under test" does the expected thing for those well defined cases.

Compare that to a programming model where you would tell the environment: "this is how I expect my function to behave"; and then that environment tries to find input that ... on which your function breaks.

If that sounds interesting to you, you should have a look into QuickCheck. Although not really "native" to Java and definitely not being "mainstream" as of now ... it might be worth giving a try.