kafka NoClassDefFoundError kafka/Kafka

Question

Regarding Apache-Kafka messaging queue.

I have downloaded Apache Kafka from the Kafka download page. I've extracted it to /opt/apache/installed/kafka-0.7.0-incubating-src.

The quickstart page says you need to start zookeeper and then start Kafka by running:
>bin/kafka-server-start.sh config/server.properties

I'm using a separate Zookeeper server, so i edited config/server.properties to point to that Zookeeper instance.

When i run Kafka, as instructed in the quickstart page, I get the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: kafka/Kafka
Caused by: java.lang.ClassNotFoundException: kafka.Kafka
        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
Could not find the main class: kafka.Kafka.  Program will exit.

I used telnet to make sure the Zookeeper instance is accessible from the machine that Kafka runs on. Everything is OK.

Why am i getting this error?

Answer 1

You must first build Kafka by running the following commands:

> ./sbt update
> ./sbt package 

Only then will Kafka be ready for use.

Answer 2

You should know that

./sbt update
./sbt package

will produce Kafka binaries for Scala 2.8.0 by default. If you need it for a different version, you need to do

./sbt "++2.9.2 update"
./sbt "++2.9.2 package"

replacing 2.9.2 with the desired version number. This will make the appropriate binaries. In general, when you switch versions, you should run

./sbt clean

to clean up the binaries from previous versions.

Actually, in addition, you might also need to perform this command

./sbt "++2.9.2 assembly-package-dependency"

This command resolves all the dependencies for running Kafka, and creates a jar that contains just these. Then the start scripts would add this to the class path and you should have all your desired classes.

Answer 3

It seems that without the SCALA_VERSION environment variable, the executable doesn't know how to load the libraries necessary. Try the following from the Kafka installation directory:

SCALA_VERSION=2.9.3 bin/kafka-server-start.sh config/server.properties

See http://kafka.apache.org/documentation.html#quickstart.

Answer 4

Just to add to the previous answer, if you're running IntelliJ, and want to run Kafka inside IntelliJ and/or step through it, make sure to run

> ./sbt idea

I spent easily half a day trying to create the IntelliJ project from scratch, and it turns out that single command was all I needed to get it working. Also, make sure you have the Scala plugin for IntelliJ installed.