Replicating messages from one Kafka topic to another kafka topic

Question

I want to make a flow from a Kafka cluster/topic in thr prod cluster into another Kafka cluster in the dev environment for scalability and regrrssion testing.

For the duck-tape solution, I cascade a Kafka consumer and producer, but my instinct tells me that there should be a better way. But, I couldn't find any good solution yet. Can anyone help me?

Answer 1

If you want to replicate data from one cluster to another then there is one kafka tool called MirrorMaker.

Kafka comes with a tool for mirroring data between Kafka clusters. The tool reads from a source cluster and writes to a destination cluster. Data will be read from topics in the source cluster and written to a topic with the same name in the destination cluster.

Here is syntax to run MirrorMaker tool:

bin/kafka-run-class.sh kafka.tools.MirrorMaker
       --consumer.config consumer.properties
       --producer.config producer.properties --whitelist my-topic

You can find this script in kafka installation directory. Here you need to provide consumer.properties of your source cluster and producer.properties of your destination cluster. You can whitelist which topics should be mirrored through --whitelist option.

You can find more information about Mirroring data between clusters

Note: MirrorMaker copies data into same topic_name in destination cluster as source cluster