java.lang.NoClassDefFoundError when running KafkaEmbedded for Unit test

Question

I have a Spring-Kafka project running Spring Boot 2.0.2. In the unit test I am trying to start a KafkaEmbedded server. I get this error.

java.lang.NoClassDefFoundError: org/apache/kafka/common/security/auth/SecurityProtocol

at kafka.utils.TestUtils$.createBrokerConfig(TestUtils.scala:222)
at kafka.utils.TestUtils.createBrokerConfig(TestUtils.scala)
at org.springframework.kafka.test.rule.KafkaEmbedded.createBrokerProperties(KafkaEmbedded.java:278)
at org.springframework.kafka.test.rule.KafkaEmbedded.before(KafkaEmbedded.java:224)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.security.auth.SecurityProtocol
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 13 more

I am aware that this is because of the dependencies mis-match. But I have added the latest versions spring-kafka-test & kafka-clients. I am aware that this class "SecurityProtocol" is in the Kafka-clients jar but adding that dependency (old & new versions) still does not solve the problem.

The following is the dependencies in pom file.

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.0.2.RELEASE</version>
    <relativePath/> <!-- lookup parent from repository -->
</parent>

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    <java.version>1.8</java.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka-test</artifactId>
        <version>2.1.6.RELEASE</version>
        <scope>test</scope>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>1.0.0</version>
        <scope>test</scope>
    </dependency>

Here is the basic Unit test that uses the EmbeddedKafka.

@RunWith(SpringRunner.class)
@SpringBootTest
public class KafkaDemoApplicationTests {

    @ClassRule
    public static KafkaEmbedded embeddedKafka = new KafkaEmbedded(1, true,
            Topology.DEMO_TOPIC);

    @Test
    public void contextLoads() {
    }

}

Answer 1

You have the wrong kafka (scala) jars on the classpath.

java.lang.NoClassDefFoundError: org/apache/kafka/common/security/auth/SecurityProtocol

SecurityProtocol was moved from org.apache.kafka.common.protocol to org.apache.kafka.common.security.auth in kafka-clients 1.0.0.

So it looks like you have (somehow) got the older kafka_2.11 jar on the CP; spring-kafka-test should bring in the right version. Perhaps you have overrides in your pom that you are not showing?