Tests pass when run individually but not when the whole test class run

Question

I have solved a topCoder problem for which all the tests pass when I run them on their own. Nonetheless when I run the whole test class some of them fail. Could you, please, help me identify the reason for this behaviour? Here is my class and my tests:

 package com.topcoder.div2.stage1;

import java.util.Arrays;

public class GameOfStones {
    private int iterations = 0;
    public int count(int[] stones){
        int result = checkEquality(stones);
        return result;
    }

    private int checkEquality(int[] stones){
        int count = 0;
        int sum = 0;
        for(int k = 0; k< stones.length;k++){
            sum += stones[k];
        }
        if(stones.length > 0) {
            for (int i = 0; i < sum; i++) {
                Arrays.sort(stones);
                if(stones[stones.length-1] != 3) {
                    int j = 0;
                    while (j < stones.length - 1) {
                        if (stones[j] == stones[j + 1]) {
                            count++;
                        }
                        j++;
                    }
                    if (count == stones.length - 1) {
                        return iterations;
                    }
                    stones[0] = stones[0] + 2;
                    stones[stones.length - 1] = stones[stones.length - 1] - 2;
                    iterations++;
                    count = 0;
                }
            }
        }
        return -1;
    }
}

Test:

package com.topcoder.div2.stage1;

import org.testng.annotations.Test;

import static org.testng.Assert.assertEquals;

public class GameOfStonesTest {
    private GameOfStones gameOfStones = new GameOfStones();

    @Test
    public void test1() {
        int expected = 0;
        int[] given = {17};

        int actual = gameOfStones.count(given);

        assertEquals(expected, actual);
    }

    @Test
    public void test2() {
        int expected = 3;
        int[] given ={7, 15, 9, 5};

        int actual = gameOfStones.count(given);
        assertEquals(actual, expected);
    }

    @Test
    public void test3() {
        int expected = -1;
        int[] given ={2, 8, 4};

        int actual = gameOfStones.count(given);
        assertEquals(actual, expected);
    }

    @Test
    public void test4() {
        int expected = -1;
        int[] given ={10, 15, 20, 12, 1, 20};

        int actual = gameOfStones.count(given);
        assertEquals(actual, expected);
    }

    @Test
    public void test5(){
        int expected = 277;
        int[] given ={17, 1, 27, 29, 13, 1, 27, 3, 19, 3, 25, 1, 11, 9, 7, 17, 31, 25, 5, 11, 31, 9,
                15, 3, 3, 3, 11, 11, 1, 41, 5, 95, 7, 3, 41, 31, 7, 13, 15, 5, 17, 3, 9, 3, 11,
                27, 1, 23, 15, 5, 43, 11, 17, 7, 1, 3, 13, 69, 3, 43, 21, 1, 25, 1, 3, 11, 5, 43,
                13, 7, 15, 1, 1, 55, 37, 9, 5, 7, 21, 3, 23, 15, 1, 9, 3, 35, 13, 17, 7, 17, 27, 5,
                9, 19, 13, 1, 1, 1, 29};
        int actual = gameOfStones.count(given);
        assertEquals(actual, expected);
    }

    @Test
     public void test6(){
        int expected = 539;
        int[] given ={1, 29, 11, 35, 57, 15, 85, 19, 5, 47, 53, 5, 63, 19, 13, 63, 27, 43, 53, 75, 67, 93, 33, 31, 47, 3,
                63, 17, 11, 53, 35, 23, 17, 45, 31, 19, 63, 75, 5, 3, 49, 19, 11, 89, 21, 69,
                71, 5, 45, 81, 31, 13, 11, 19, 7, 99, 33, 63, 19, 57, 73, 29, 35, 9, 47,
                1, 17, 7, 13, 31, 5, 85, 95, 23, 45, 65, 63, 41, 81, 33, 45, 1, 15,
                45, 19, 87, 51, 7, 13, 39, 1, 59, 29, 35, 1, 43};
        int actual = gameOfStones.count(given);
        assertEquals(actual, expected);
    }

    @Test
    public void test7() {
        int expected = 0;
        int[] given ={100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100,
                100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100,
                100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100,
                100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100,
                100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100,
                100, 100};
        int actual = gameOfStones.count(given);
        assertEquals(actual, expected);
    }

    @Test
    public void test8() {
        int expected = 11;
        int[] given ={3, 5, 21, 31};
        int actual = gameOfStones.count(given);
        assertEquals(actual, expected);
    }

    @Test
    public void test9() {
        int expected = 13;
        int[] given ={44, 6, 46};
        int actual = gameOfStones.count(given);
        assertEquals(actual, expected);
    }

}

P.S if you know any suggestions of improving the solution you are more than welcome to include them in your answer.

Answer 1

You are sharing a single instance of the class under test across all tests. I'd remove the initial assignment and add this:

private GameOfStones gameOfStones; // Don't create an instance here

@BeforeMethod
public void setUp() {
    gameOfStones = new GameOfStones();
}

... which will use a new instance for each test. Good practice would also be to clean up after each test:

@AfterMethod
public void tearDown() {
    gameOfStones = null;
}

In the example given here, fixing the class scoped variable causing the problem to be method scoped instead would also fix the issue, but as the software under test gets more complex it's good to start doing proper test set up and tear down.

Answer 2

I had same issue. I needed to mock a logger, which was a static field. So eventually class loader creates only a single instance of the static field during the first invocation of a class under the test and disregards all further mocking and stubbing. When run separately, test was green, because the logger was initialized and loaded as expected, but when run all together, with other test methods, it got initialized as a concrete object, not a mock. Workaround:

• create @BeforeClass method to ensure that the right instance of static field will be created in the first place:

    @BeforeClass
    public static void setupBeforeClass() {
      PowerMockito.mockStatic(LoggerFactory.class);
      loggerMock = mock(Logger.class);
      when(LoggerFactory.getLogger(any(Class.class))).thenReturn(loggerMock);
   }

• Interactions on the mock are getting accumulated from different test executions. Therefore, to be sure that you get a clean instance of the mock on each test method execution, reset the mock whether in @Before or @After method:

      @Before
      public void setup() {

        // Reset interactions on the mocked logger
        Mockito.reset(loggerMock);

      }

Note, that in my example I used PowerMock so you need a corresponding runner @RunWith(PowerMockRunner.class) and @PrepareForTest({LoggerFactory.class, MyClass.class)} statements.