what is the best approach to keep two kafka clusters in Sync

Question

I have to setup two kafka clusters in two different data centers (DCs), which have same topics and configuration. the reason is that the connectivity between two data centers is nasty we cannot create a global one.

We are having producers and consumers to publish and subcribe to the topics of each DC.

the problem is that I need to keep both clusters in sync. Lets say: all messages are written to the first DC should be eventually replicated to the second, and otherway around.

I am evaluation the kafka MirrorMaker tool by creating the Mirror by consuming messages of the first and procuding messages to the second one. However it is also requried to replicate data from the second to the first because writing data is both allowed in two clusters.

I dont think the Kafka MirrorMaker tool is fit to our case. Appricate any suggestion?

Thanks in advance.

Answer 1

Depending on your exact requirements, you can use MirrorMaker for your use case.

One option would be to just have two separate topics, lets call them topic1 on cluster 1 and topic2 on cluster 2. All your producing threads write to the "local" topic and you use mirrormaker to replicate this topic to the remote cluster.

For your consumers, you simply subscribe to both topics on whatever cluster is closest to you, that way you will get all records that were written on either cluster. I have created an illustration that hopefully helps:

Alternatively, you could create aggregation topics on both clusters and use MirrorMaker to replicate data into this topic, this would enable you to have all data in one topic for consumption. You would have duplicate data on the same cluster this way, but you could take care of this by lower retention settings on the input topic. Again, hopefully the following picture helps to explains my thinking: In order for this to work, you will need to configure MirrorMaker to replicate a topic into a topic with a different name, which is not a standard thing for it to do, I have written a small blog post on how to do this, if you want to investigate this option further.