Parallel test with shared mock

Question

I'm writing a unit test class (using testng) that has mocked member variables (using Mockito) and running the tests in parallel. I initially set up the expected mock in an @BeforeClass method, and in each test case I break something by creating a Mockito.when for each exceptional case.

What I'm seeing (unsurprisingly) is that these tests aren't independent; the Mockito.when in one test case affects the others. I noticed that I could be set up the mocks before each test, and I changed the @BeforeClass to @BeforeMethod. I still didn't expect these to pass consistently, as the tests are all still operating on the same shared mock object at the same time. However, all the tests started passing consistently. My question is "why"? Will this eventually fail? No matter what I do (Thread.sleep, etc) I can't reproduce a failure.

Is using @BeforeMethod enough to make these tests independent? If so, can anyone explain why?

Example code below:

public class ExampleTest {
    @Mock
    private List<String> list;

    @BeforeClass // Changing to @BeforeMethod works for some reason
    public void setup() throws NoSuchComponentException, ADPRuntimeException {
        MockitoAnnotations.initMocks(this);
        Mockito.when(list.get(0)).thenReturn("normal");
    }

    @Test
    public void testNormalCase() throws InterruptedException {
        assertEquals(list.get(0), "normal"); // Fails with expected [normal] but found [exceptional]
    }

    @Test
    public void testExceptionalCase() throws InterruptedException {
        Mockito.when(list.get(0)).thenReturn("exceptional");
        assertEquals(list.get(0), "exceptional");
    }
}

Answer 1

The problem here is that TestNG creates one instance of your test class ExampleTest and this is the instance that is used by both of your @Test methods.

So when you used @BeforeClass, you would have random failures with testNormalCase() if testExceptionalCase() ran first and altered the state of your test class.

When you changed your annotation to be @BeforeMethod, it would cause the setup to be executed right before every @Test method was executed.

So the setup would fix the state for testNormalCase() which is why it would pass, and since testExceptionalCase() was internally altering the state using Mockito.when() and then running assertions, it would pass all the time as well.

But there's one scenario wherein your setup will still fail, viz., when you use parallel="methods" attribute in your <suite> tag within your TestNG suite xml file i.e., when you configure TestNG and instruct it to run every @Test method in parallel.

In that case, the Mockito.when() within testExceptionalCase() will affect the shared state [ since you are using this in a shared manner amongst all your @Test methods ] causing testNormalCase() to fail randomly.

To fix this, I would suggest that you do the following :

• Don't share this between your @Test methods, but house it separately outside of your test class i.e., house all the data members of your test class in a separate pojo which would be mocked rather than mocking this.
• Use a ThreadLocal to store the state which is being mocked by Mockito.when() and then run assertions on the ThreadLocal from within your @Test methods.