Setting up Apache Kafka for developer/integration test environment

Question

I am looking for advice and pointers on how to set up and run Apache Kafka on a developer & integration test environment.

I would like to avoid having to manually download and set up the whole package on each of the developers' machines and also find an easy way to start the nodes automatically when for integration tests.

Is there some sort of embedded Kafka for dev/integration test purposes (think of the H2 parallel for java developers wanting to avoid a full-fledged RDBMS)?

Do I have to resort to some sort of Vagrant solution? By the way, I have found the following interesting Gist: https://gist.github.com/svanellewee/8d978db827a860186569 but it requires setting up Vagrant+VirtualBox...

Answer 1

Is there some sort of embedded Kafka for dev/integration test purposes

Kafka actually includes test facilities that lets you run embedded Kafka clusters as well as unit test facilities (starting with Kafka v0.10.2.0 from 2017). See the documentation at: https://kafka.apache.org/documentation/streams/developer-guide/testing.html

For example, you can add the relevant maven artifacts to your build via:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-streams-test-utils</artifactId>
    <!-- Replace with desired Kafka version. -->
    <version>2.2.0</version>
    <scope>test</scope>
</dependency>

Documentation is available at: https://kafka.apache.org/documentation/streams/developer-guide/testing.html

The following pointers will be helpful for writing tests:

• https://github.com/confluentinc/kafka-streams-examples -- kafka-streams sub-folder includes many end-to-end integration tests that spin up embedded Kafka clusters as well as, in some cases, Confluent schema registry. These should be a good starting point for your own tests.
• https://github.com/apache/kafka/tree/trunk/streams/src/test/java/org/apache/kafka/streams/integration/utils -- similar test facilities as in the previous link, but fewer included examples

I would like to avoid having to manually download and set up the whole package on each of the developers' machines and also find an easy way to start the nodes automatically when for integration tests.

The above might be the easiest if your needs are covered, because it blends well into an existing development/testing process with tools like maven, gradle, or sbt.

However, if you need to perform more advanced testing -- such as intentionally taking down Kafka brokers mid-way to validate that your app survives partial outages, for example -- you might want to take a look at:

• https://github.com/apache/kafka/tree/trunk/tests (system and integration tests, using e.g. docker)
• https://github.com/apache/kafka/tree/trunk/vagrant
• https://github.com/confluentinc/ducktape

Answer 2

I think KafkaUnit is what comes closest to H2 here. You can find it here (https://github.com/chbatey/kafka-unit). Or just use via

<dependency>
    <groupId>info.batey.kafka</groupId>
    <artifactId>kafka-unit</artifactId>
    <version>0.6</version>
</dependency>

Works for unit/integration-tests all in one JVM, similar to Curator Testingserver.