Check that JUnit Extension throws specific Exception

Question

Suppose I develop an extension which disallows test method names to start with an uppercase character.

public class DisallowUppercaseLetterAtBeginning implements BeforeEachCallback {

    @Override
    public void beforeEach(ExtensionContext context) {
        char c = context.getRequiredTestMethod().getName().charAt(0);
        if (Character.isUpperCase(c)) {
            throw new RuntimeException("test method names should start with lowercase.");
        }
    }
}

Now I want to test that my extension works as expected.

@ExtendWith(DisallowUppercaseLetterAtBeginning.class)
class MyTest {

    @Test
    void validTest() {
    }

    @Test
    void TestShouldNotBeCalled() {
        fail("test should have failed before");
    }
}

How can I write a test to verify that the attempt to execute the second method throws a RuntimeException with a specific message?

Answer 1

Another approach could be to use the facilities provided by the new JUnit 5 - Jupiter framework.

I put below the code which I tested with Java 1.8 on Eclipse Oxygen. The code suffers from a lack of elegance and conciseness but could hopefully serve as a basis to build a robust solution for your meta-testing use case.

Note that this is actually how JUnit 5 is tested, I refer you to the unit tests of the Jupiter engine on Github.

public final class DisallowUppercaseLetterAtBeginningTest { 
    @Test
    void testIt() {
        // Warning here: I checked the test container created below will
        // execute on the same thread as used for this test. We should remain
        // careful though, as the map used here is not thread-safe.
        final Map<String, TestExecutionResult> events = new HashMap<>();

        EngineExecutionListener listener = new EngineExecutionListener() {
            @Override
            public void executionFinished(TestDescriptor descriptor, TestExecutionResult result) {
                if (descriptor.isTest()) {
                    events.put(descriptor.getDisplayName(), result);
                }
                // skip class and container reports
            }

            @Override
            public void reportingEntryPublished(TestDescriptor testDescriptor, ReportEntry entry) {}
            @Override
            public void executionStarted(TestDescriptor testDescriptor) {}
            @Override
            public void executionSkipped(TestDescriptor testDescriptor, String reason) {}
            @Override
            public void dynamicTestRegistered(TestDescriptor testDescriptor) {}
        };

        // Build our test container and use Jupiter fluent API to launch our test. The following static imports are assumed:
        //
        // import static org.junit.platform.engine.discovery.DiscoverySelectors.selectClass
        // import static org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder.request

        JupiterTestEngine engine = new JupiterTestEngine();
        LauncherDiscoveryRequest request = request().selectors(selectClass(MyTest.class)).build();
        TestDescriptor td = engine.discover(request, UniqueId.forEngine(engine.getId())); 

        engine.execute(new ExecutionRequest(td, listener, request.getConfigurationParameters()));

        // Bunch of verbose assertions, should be refactored and simplified in real code.
        assertEquals(new HashSet<>(asList("validTest()", "TestShouldNotBeCalled()")), events.keySet());
        assertEquals(Status.SUCCESSFUL, events.get("validTest()").getStatus());
        assertEquals(Status.FAILED, events.get("TestShouldNotBeCalled()").getStatus());

        Throwable t = events.get("TestShouldNotBeCalled()").getThrowable().get();
        assertEquals(RuntimeException.class, t.getClass());
        assertEquals("test method names should start with lowercase.", t.getMessage());
}

Though a little verbose, one advantage of this approach is it doesn't require mocking and execute the tests in the same JUnit container as will be used later for real unit tests.

With a bit of clean-up, a much more readable code is achievable. Again, JUnit-Jupiter sources can be a great source of inspiration.

Answer 2

If the extension throws an exception then there's not much a @Test method can do since the test runner will never reach the @Test method. In this case, I think, you have to test the extension outside of its use in the normal test flow i.e. let the extension be the SUT. For the extension provided in your question, the test might be something like this:

@Test
public void willRejectATestMethodHavingANameStartingWithAnUpperCaseLetter() throws NoSuchMethodException {
    ExtensionContext extensionContext = Mockito.mock(ExtensionContext.class);
    Method method = Testable.class.getMethod("MethodNameStartingWithUpperCase");

    Mockito.when(extensionContext.getRequiredTestMethod()).thenReturn(method);

    DisallowUppercaseLetterAtBeginning sut = new DisallowUppercaseLetterAtBeginning();

    RuntimeException actual =
            assertThrows(RuntimeException.class, () -> sut.beforeEach(extensionContext));
    assertThat(actual.getMessage(), is("test method names should start with lowercase."));
}

@Test
public void willAllowTestMethodHavingANameStartingWithAnLowerCaseLetter() throws NoSuchMethodException {
    ExtensionContext extensionContext = Mockito.mock(ExtensionContext.class);
    Method method = Testable.class.getMethod("methodNameStartingWithLowerCase");

    Mockito.when(extensionContext.getRequiredTestMethod()).thenReturn(method);

    DisallowUppercaseLetterAtBeginning sut = new DisallowUppercaseLetterAtBeginning();

    sut.beforeEach(extensionContext);

    // no exception - good enough
}

public class Testable {
    public void MethodNameStartingWithUpperCase() {

    }
    public void methodNameStartingWithLowerCase() {

    }
}

However, your question suggests that the above extension is only an example so, more generally; if your extension has a side effect (e.g. sets something in an addressable context, populates a System property etc) then your @Test method could assert that this side effect is present. For example:

public class SystemPropertyExtension implements BeforeEachCallback {

    @Override
    public void beforeEach(ExtensionContext context) {
        System.setProperty("foo", "bar");
    }
}

@ExtendWith(SystemPropertyExtension.class)
public class SystemPropertyExtensionTest {

    @Test
    public void willSetTheSystemProperty() {
        assertThat(System.getProperty("foo"), is("bar"));
    }
}

This approach has the benefit of side stepping the potentially awkward setup steps of: creating the ExtensionContext and populating it with the state required by your test but it may come at the cost of limiting the test coverage since you can really only test one outcome. And, of course, it is only feasible if the extension has a side effect which can be evaulated in a test case which uses the extension.

So, in practice, I suspect you might need a combination of these approaches; for some extensions the extension can be the SUT and for others the extension can be tested by asserting against its side effect(s).