How to automate Kafka Testing

Question

We have developed a system using kafka to queue the data and later consume that data to place orders for users.
We have tested certain things manually, but now our aim is automate the process.
Is there any client available to test it? I found out ways to Unit test it using kafka client itself, but my aim is to test the system as whole.

EDIT: our purpose is just API testing i.e., just the back-end, not the UI

Answer 1

You can start Kafka programmatically in your integration test, Kafka uses Zookeeper so firsly look at Zookeeper TestingServer - instance of this class creates and starts the Zk server using the given port.

Next look at KafkaServerStartable.scala, you have to provide configuration that points to your in memory Zk server and invoke startup() method, here is some code:

import kafka.server.KafkaConfig; 
import kafka.server.KafkaServerStartable;
import java.util.Properties;

public KafkaTest() {
    Properties properties = createProperties();
    KafkaConfig kafkaConfig = new KafkaConfig(properties);
    KafkaServerStartable kafka = new KafkaServerStartable(kafkaConfig);
    kafka.startup();
}

Hope these help:)

Answer 2

You can go for integration-testing or end-to-end testing by bringing up Kafka in a docker container. If you use Apache kafka-clients:2.1.0, then you don't need to deal with ZooKeeper at the API level while producing or consuming the records.

Dockerizing Kafka, and testing helps to cover the scenarios in a single node as well as multi-node Kafka cluster. This way you don't have to test against Mock/In-Memory Kafka once, then real Kafka later. This can be done using TestContainers.

If you have too many test scenarios to cover, you can go for Kafka Declarative Testing like docker-compose style, by which you can eliminate the Kafka client API coding.

Checkout some handy examples here for validating produce and consume.

TestContainers project also supports docker-compose.