Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Kafka topic alias

Tags:

apache-kafka

Is it possible to create an alias of a topic name?

Or, put another way...

If a user writes to topic examplea is it possible to override that at the broker so they actually write to topic exampleb?

alternatively, if the topic was actually written as examplea, but the consumer can refer to it as exampleb.

I'm thinking it could probably be achieved using small hack at the broker where it replies to metadata requests, but I'd rather not if it can be done in some standard way.

like image 319
Graham Avatar asked Jun 26 '16 09:06

Graham


People also ask

Can a Kafka broker have multiple topics?

We can create many topics in Apache Kafka, and it is identified by unique name. Topics are split into partitions, each partition is ordered and messages with in a partitions gets an id called Offset and it is incremental unique id.

What is Kafdrop?

Kafdrop is a web UI for viewing Kafka topics and browsing consumer groups. The tool displays information such as brokers, topics, partitions, consumers, and lets you view messages. This project is a reboot of Kafdrop 2. x, dragged kicking and screaming into the world of JDK 11+, Kafka 2.

Can a Kafka producer write to multiple topics?

Kafka is able to seamlessly handle multiple producers that are using many topics or the same topic. The consumer subscribes to one or more topics and reads the messages. The consumer keeps track of which messages it has already consumed by keeping track of the offset of messages.


1 Answers

Aliases are not natively supported in Kafka.

One workaround could be to produce to examplea and have a consumer/producer pair that consumers from examplea and produces to exampleb. The consumer/producer pair could be written with Kafka clients, as a connector in Connect, as a MirrorMaker instance (though you'll need to modify it to change the topic name), or as a Kafka Streams job. Note that the messages will appear in exampleb slightly after examplea because they're being copied after being written.

like image 178
alexlod Avatar answered Nov 03 '22 20:11

alexlod