Kafka Mirror Maker failing to replicate __consumer_offset topic

Question

I am trying to make use of mirror maker to replicate __consumer_offsets topic along with other topics.

It is giving error as mentioned below.

[2018-10-24 16:16:03,802] ERROR Error when sending message to topic __consumer_offsets with key: 16 bytes, value: 445 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.InvalidTopicException: The request attempted to perform an operation on an invalid topic. [2018-10-24 16:16:03,828] ERROR Error when sending message to topic __consumer_offsets with key: 29 bytes, value: 754 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.InvalidTopicException: The request attempted to perform an operation on an invalid topic.

Is there any way to resolve this?

In one of the confluent presentation in SlideShare on slide21, it mentioned about replicating offset topic in multiple datacenters. Can someone tell me the possible way to achieve the same?

Or there is any other better way for backup and restore policy for Kafka.

Answer 1

Add this in your consumer.config:

exclude.internal.topics=false

And add this in your producer.config:

client.id=__admin_client

Reason being, Producers cannot write to internal topics unless you declare client.id=__admin_client which is used by AdminClient.scala. Found here: https://issues.apache.org/jira/browse/KAFKA-6524