What is the best way to test that a spring application context fails to start?

Question

I use the spring-boot-starter-web and spring-boot-starter-test.

Let's say I have a class for binding configuration properties:

@ConfigurationProperties(prefix = "dummy")
public class DummyProperties {

    @URL
    private String url;

    // getter, setter ...

}

Now I want to test that my bean validation is correct. The context should fail to start (with a specfic error message) if the property dummy.value is not set or if it contains an invalid URL. The context should start if the property contains a valid URL. (The test would show that @NotNull is missing.)

A test class would look like this:

@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(classes = MyApplication.class)
@IntegrationTest({ "dummy.url=123:456" })
public class InvalidUrlTest {
    // my test code
}

This test would fail because the provided property is invalid. What would be the best way to tell Spring/JUnit: "yep, this error is expected". In plain JUnit tests I would use the ExpectedException.

Answer 1

Why is that an integration test to begin with? Why are you starting a full blown Spring Boot app for that?

This looks like unit testing to me. That being said, you have several options:

• Don't add @IntegrationTest and Spring Boot will not start a web server to begin with (use @PropertySource to pass value to your test but it feels wrong to pass an invalid value to your whole test class)
• You can use spring.main.web-environment=false to disable the web server (but that's silly given the point above)
• Write a unit test that process that DummyProperties of yours. You don't even need to start a Spring Boot application for that. Look at our own test suite

I'd definitely go with the last one. Maybe you have a good reason to have an integration test for that?

Answer 2

The best way to test Spring application context is to use ApplicationContextRunner

It is described in Spring Boot Reference Documentation:
https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-developing-auto-configuration.html#boot-features-test-autoconfig

And there is a quick guide about it:
https://www.baeldung.com/spring-boot-context-runner

Sample usage

private static final String POSITIVE_CASE_CONFIG_FILE =  
"classpath:some/path/positive-case-config.yml";
private static final String NEGATIVE_CASE_CONFIG_FILE =  
"classpath:some/path/negative-case-config.yml";

@Test
void positiveTest() {
  ApplicationContextRunner contextRunner = new ApplicationContextRunner()
    .withInitializer(new ConfigDataApplicationContextInitializer())//1
    .withInitializer(new ConditionEvaluationReportLoggingListener(LogLevel.DEBUG))//2
    .withUserConfiguration(MockBeansTestConfiguration.class)//3
    .withPropertyValues("spring.config.location=" + POSITIVE_CASE_CONFIG_FILE)//4
    .withConfiguration(AutoConfigurations.of(BookService.class));//5
  contextRunner
    .run((context) -> {
      Assertions.assertThat(context).hasNotFailed();//6
    });
}

@Test
void negativeTest() {
  ApplicationContextRunner contextRunner = new ApplicationContextRunner()
    .withInitializer(new ConfigDataApplicationContextInitializer())//1
    .withInitializer(new ConditionEvaluationReportLoggingListener(LogLevel.DEBUG))//2
    .withUserConfiguration(MockBeansTestConfiguration.class)//3
    .withPropertyValues("spring.config.location=" + NEGATIVE_CASE_CONFIG_FILE)//4
    .withConfiguration(AutoConfigurations.of(BookService.class));//5
  contextRunner
    .run((context) -> {
      assertThat(context)
        .hasFailed();
      assertThat(context.getStartupFailure())
        .isNotNull();
      assertThat(context.getStartupFailure().getMessage())
        .contains("Some exception message");
      assertThat(extractFailureCauseMessages(context))
        .contains("Cause exception message");
    });
}

private List<String> extractFailureCauseMessages(AssertableApplicationContext context) {
  var failureCauseMessages = new ArrayList<String>();
  var currentCause = context.getStartupFailure().getCause();
  while (!Objects.isNull(currentCause)) {//7
    failureCauseMessages.add(currentCause.getMessage());
    currentCause = currentCause.getCause();
  }
  return failureCauseMessages;
}

Explanation with examples of similar definitions from Junit5 Spring Boot Test Annotations:

1. Triggers loading of config files like application.properties or application.yml
2. Logs ConditionEvaluationReport using given log level when application context fails
3. Provides class that specifies mock beans, ie. we have @Autowired BookRepository in our BookService and we provide mock BookRepository in MockBeansTestConfiguration. Similar to @Import({MockBeansTestConfiguration.class}) in test class and @TestConfiguration in class with mock beans in normal Junit5 Spring Boot Test
4. Equivalent of @TestPropertySource(properties = { "spring.config.location=" + POSITIVE_CASE_CONFIG_FILE})
5. Triggers spring auto configuration for given class, not direct equivalent, but it is similar to using @ContextConfiguration(classes = {BookService.class}) or @SpringBootTest(classes = {BookService.class}) together with @Import({BookService.class}) in normal test
6. Assertions.class from AssertJ library, there should be static import for Assertions.assertThat, but I wanted to show where this method is from
7. There should be static import for Objects.isNull, but I wanted to show where this method is from

MockBeansTestConfiguration class:

@TestConfiguration
public class MockBeansTestConfiguration {
  private static final Book SAMPLE_BOOK = Book.of(1L, "Stanisław Lem", "Solaris", "978-3-16-148410-0");

  @Bean
  public BookRepository mockBookRepository() {
    var bookRepository = Mockito.mock(BookRepository.class);//1
    Mockito.when(bookRepository.findByIsbn(SAMPLE_BOOK.getIsbn()))//2
           .thenReturn(SAMPLE_BOOK);
    return bookRepository;
  }
}

Remarks:
1,2. There should be static import, but I wanted to show where this method is from