Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can single Kafka producer produce messages to multiple topics and how?

I am just exploring Kafka, currently i am using One producer and One topic to produce messages and it is consumed by one Consumer. very simple.

I was reading the Kafka page, the new Producer API is thread-safe and sharing single instance will improve the performance.

Does it mean i can use single Producer to publish messages to multiple topics?

like image 264
Shankar Avatar asked Aug 23 '16 06:08

Shankar


People also ask

Can Kafka producer produce to multiple topics?

Kafka is able to seamlessly handle multiple producers that are using many topics or the same topic. The consumer subscribes to one or more topics and reads the messages.

Can a producer publish to multiple topics?

Does it mean i can use single Producer to publish messages to multiple topics? Yes, see the docs at kafka.apache.org/documentation.html#producerapi.

Can a Kafka producer write to multiple partitions?

A Kafka producer can write to different partitions in parallel, which generally means that it can achieve higher levels of throughput.

Can Kafka have multiple topics?

Multi-Topic ConsumersWe may have a consumer group that listens to multiple topics. If they have the same key-partitioning scheme and number of partitions across two topics, we can join data across the two topics.


1 Answers

Never tried it myself, but I guess you can. Since the code for producer and sending the record is (from here https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html):

Producer<String, String> producer = new KafkaProducer<>(props);
 for(int i = 0; i < 100; i++)
     producer.send(new ProducerRecord<String, String>("my-topic", Integer.toString(i), Integer.toString(i)));

So, I guess, if you just write different topics in the ProducerRecord, than it should be possible.

Also, here http://kafka.apache.org/081/documentation.html#producerapi it explicitly says that you can use a method send(List<KeyedMessage<K,V>> messages) to write into multiple topics.

like image 162
RadioLog Avatar answered Sep 22 '22 16:09

RadioLog