Tests run under JUnit 4 but not JUnit 5 — Compiles clean, but 0 tests execute

Question

Anyone can easily reproduce this problem in a couple minutes.

Basic Maven quickstart project

With IntelliJ 2018.3 and Maven 3.6.0, I create a brand new project using the Maven archetype maven-archetype-quickstart version 1.4.

Java 11

In the POM file of the new project, I change properties for maven.compiler.source and maven.compiler.target from 1.7 to 11, for the Java 11.0.2 I am currently using, Zulu from Azul Systems.

<properties>
  <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  <maven.compiler.source>11</maven.compiler.source>
  <maven.compiler.target>11</maven.compiler.target>
</properties>

On the Maven panel of IntelliJ, I run the clean and install Lifecycle events.

Test runs in JUnit 4

As part of install, the tests are run. This quickstart archetype comes with one single test that asserts true.

The results appear in the Run panel of IntelliJ.

[INFO] Running work.basil.example.AppTest

[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.026 s - in work.basil.example.AppTest

So I know the test executed.

JUnit 5, not 4

This is all good. Now let's upgrade to JUnit 5, to see the problem.

In the POM, I change the JUnit dependency from this:

<dependencies>
  <dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
    <version>4.11</version>
    <scope>test</scope>
  </dependency>
</dependencies>

…to this:

<dependencies>
  <!--JUnit 5-->
  <dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-api</artifactId>
    <version>5.3.2</version>
    <scope>test</scope>
  </dependency>
</dependencies>

Jupiter imports (no vintage tests)

The compiler complains about my AppTest.java file. So I change the import statements there to use the jupiter packages. I only want to run JUnit 5 tests in my new greedfield project, with no need for vintage JUnit 4 tests. So the imports change from this:

import static org.junit.Assert.assertTrue;
import org.junit.Test;

…to this:

import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertTrue;

Then I execute the Maven > Lifecycle > clean & install.

…and voilà, the problem: Our test is not executed. The report seen in the Run panel of IntelliJ:

[INFO] Running work.basil.example.AppTest

[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s - in work.basil.example.AppTest

➥ Why does JUnit 5 fail to run the very same test that JUnit 4 happily ran?

Update surefire plugin

I suspect the Maven Surefire Plugin needs to be updated. So in the POM I change this:

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.22.1</version>
</plugin>

…to this:

<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.0.0-M3</version>
</plugin>

Another clean & install. But no better, still runs 0 tests.

[INFO] Running work.basil.example.AppTest

[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 s - in work.basil.example.AppTest

Entire POM

Here is my entire POM file.

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>work.basil.example</groupId>
  <artifactId>tester</artifactId>
  <version>1.0-SNAPSHOT</version>

  <name>tester</name>
  <!-- FIXME change it to the project's website -->
  <url>http://www.example.com</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>11</maven.compiler.source>
    <maven.compiler.target>11</maven.compiler.target>
  </properties>

  <dependencies>
    <!--JUnit 5-->
    <dependency>
      <groupId>org.junit.jupiter</groupId>
      <artifactId>junit-jupiter-api</artifactId>
      <version>5.3.2</version>
      <scope>test</scope>
    </dependency>
  </dependencies>

  <build>
    <pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
      <plugins>
        <!-- clean lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#clean_Lifecycle -->
        <plugin>
          <artifactId>maven-clean-plugin</artifactId>
          <version>3.1.0</version>
        </plugin>
        <!-- default lifecycle, jar packaging: see https://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_jar_packaging -->
        <plugin>
          <artifactId>maven-resources-plugin</artifactId>
          <version>3.0.2</version>
        </plugin>
        <plugin>
          <artifactId>maven-compiler-plugin</artifactId>
          <version>3.8.0</version>
        </plugin>
        <plugin>
          <artifactId>maven-surefire-plugin</artifactId>
          <version>3.0.0-M3</version>
        </plugin>
        <plugin>
          <artifactId>maven-jar-plugin</artifactId>
          <version>3.0.2</version>
        </plugin>
        <plugin>
          <artifactId>maven-install-plugin</artifactId>
          <version>2.5.2</version>
        </plugin>
        <plugin>
          <artifactId>maven-deploy-plugin</artifactId>
          <version>2.8.2</version>
        </plugin>
        <!-- site lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#site_Lifecycle -->
        <plugin>
          <artifactId>maven-site-plugin</artifactId>
          <version>3.7.1</version>
        </plugin>
        <plugin>
          <artifactId>maven-project-info-reports-plugin</artifactId>
          <version>3.0.0</version>
        </plugin>
      </plugins>
    </pluginManagement>
  </build>
</project>

JUnit libraries

After doing a Maven clean & install, two JUnit libraries appear: junit-jupiter-api and junit-platform-commons.

Other versions of JUnit 5

I tried the following versions in my junit-jupiter-api dependency:

• 5.0.0-M1
• 5.1.1
• 5.3.0
• 5.3.2
• 5.4.0-M1

On each attempt, I ran a Maven clean & install. No better. Each of those versions reported Tests run: 0.

Do not blame maven-archetype-quickstart

I actually discovered this problem in a much different project using an entirely different Maven archetype.

To nail down this buggy JUnit 5 behavior, I tried a new fresh project using the very simple maven-archetype-quickstart. I found the very same behavior: Everything compiles, the test harness in running, but no tests are executed under JUnit 5.

Answer 1

tl;dr

For JUnit 5 version 5.4.0-M1 or later, specify the new single Maven artifact junit-jupiter “Aggregator” in your POM.

<!--JUnit 5-->
<!-- https://mvnrepository.com/artifact/org.junit.jupiter/junit-jupiter -->
<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter</artifactId>
    <version>5.4.0-M1</version>
</dependency>

For earlier versions, specify at least these two artifacts: junit-jupiter-api & junit-jupiter-engine.

JUnit 5 yokes multiple testing frameworks

From what I can gather, JUnit 5 has been re-architected to be a yoke for multiple testing frameworks. These testing systems include JUnit 4 “vintage” tests, the new JUnit 5 tests (new syntax for tests, with new annotations & methods), and others such as Specsy, Spek, Cucumber, Drools Scenario, jqwik, and more that implement the TestEngine interface.

Apparently the junit-jupiter-api artifact is only the outer yoke. You must also specify one or more TestEngine implementations to actually run tests. For example, to run the vintage JUnit 4 tests, you need the VintageTestEngine implementations, or to run JUNit 5 tests you need the JupiterTestEngine implementation.

So to run your JUnit 5 tests, you must specify a JupiterTestEngine implementation in your Maven POM with the junit-jupiter-engine artifact.

See the JUnit 5 manual, specifically the section Configuring Test Engines.

See this presentation by Marc Philipp, with a diagram showing JUnit 5 as a platform having (A) a core for IDE/build tools with (B) pluggable test-writing frameworks for programmers authoring tests.

junit-jupiter-engine

As seen on this sample, add a second JUnit-related dependency for the JUNit Jupiter Engine. The documentation for this artifact says simply: “JUnit Jupiter test engine implementation, only required at runtime.”.

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-engine</artifactId>
    <version>5.4.0-M1</version>
    <scope>test</scope>
</dependency>

Simply adding that one dependency to the project shown in your Question will see your tests run.

[INFO] Running work.basil.example.AppTest

[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 s - in work.basil.example.AppTest

---

junit-jupiter-params

That same sample shows also a third JUnit dependency, for JUnit Jupiter Params. While not needed to make your example test run, it may serve other purposes. Apparently related to Parameterized Tests.

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-params</artifactId>
    <version>5.4.0-M1</version>
    <scope>test</scope>
</dependency>

That makes a total of 3 JUnit dependencies.

<!--JUnit 5-->
<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-api</artifactId>
    <version>5.4.0-M1</version>
    <scope>test</scope>
</dependency>

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-params</artifactId>
    <version>5.4.0-M1</version>
    <scope>test</scope>
</dependency>

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-engine</artifactId>
    <version>5.4.0-M1</version>
    <scope>test</scope>
</dependency>

Your same POM file, now updated to all 3 of these JUnit dependencies.

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns = "http://maven.apache.org/POM/4.0.0"
         xmlns:xsi = "http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation = "http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>work.basil.example</groupId>
    <artifactId>tester</artifactId>
    <version>1.0-SNAPSHOT</version>

    <name>tester</name>
    <!-- FIXME change it to the project's website -->
    <url>http://www.example.com</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
    </properties>

    <dependencies>

        <!--JUnit 5-->
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>5.4.0-M1</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-params</artifactId>
            <version>5.4.0-M1</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>5.4.0-M1</version>
            <scope>test</scope>
        </dependency>

    </dependencies>

    <build>
        <pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
            <plugins>
                <!-- clean lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#clean_Lifecycle -->
                <plugin>
                    <artifactId>maven-clean-plugin</artifactId>
                    <version>3.1.0</version>
                </plugin>
                <!-- default lifecycle, jar packaging: see https://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_jar_packaging -->
                <plugin>
                    <artifactId>maven-resources-plugin</artifactId>
                    <version>3.0.2</version>
                </plugin>
                <plugin>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>3.8.0</version>
                </plugin>
                <plugin>
                    <artifactId>maven-surefire-plugin</artifactId>
                    <version>3.0.0-M3</version>
                </plugin>
                <plugin>
                    <artifactId>maven-jar-plugin</artifactId>
                    <version>3.0.2</version>
                </plugin>
                <plugin>
                    <artifactId>maven-install-plugin</artifactId>
                    <version>2.5.2</version>
                </plugin>
                <plugin>
                    <artifactId>maven-deploy-plugin</artifactId>
                    <version>2.8.2</version>
                </plugin>
                <!-- site lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#site_Lifecycle -->
                <plugin>
                    <artifactId>maven-site-plugin</artifactId>
                    <version>3.7.1</version>
                </plugin>
                <plugin>
                    <artifactId>maven-project-info-reports-plugin</artifactId>
                    <version>3.0.0</version>
                </plugin>
            </plugins>
        </pluginManagement>
    </build>
</project>

junit-jupiter artifact

Version 5.4.0 of JUnit 5 brings a new Maven artifact, junit-jupiter titled JUnit Jupiter (Aggregator). The word “aggregator* apparently refers to it bundling a few of the commonly-used JUnit 5 artifacts in Maven, for our programming convenience.

Adding this one single dependency in your POM gets you 8 libraries in your project.

<!--JUnit 5-->
<!-- https://mvnrepository.com/artifact/org.junit.jupiter/junit-jupiter -->
<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter</artifactId>
    <version>5.4.0-M1</version>
</dependency>