I am just exploring Kafka
, currently i am using One producer
and One topic to produce messages and it is consumed by one Consumer
. very simple.
I was reading the Kafka page, the new Producer API is thread-safe
and sharing single instance will improve the performance.
Does it mean i can use single Producer to publish messages to multiple topics?
Kafka is able to seamlessly handle multiple producers that are using many topics or the same topic. The consumer subscribes to one or more topics and reads the messages.
Does it mean i can use single Producer to publish messages to multiple topics? Yes, see the docs at kafka.apache.org/documentation.html#producerapi.
A Kafka producer can write to different partitions in parallel, which generally means that it can achieve higher levels of throughput.
Multi-Topic ConsumersWe may have a consumer group that listens to multiple topics. If they have the same key-partitioning scheme and number of partitions across two topics, we can join data across the two topics.
Never tried it myself, but I guess you can. Since the code for producer and sending the record is (from here https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html):
Producer<String, String> producer = new KafkaProducer<>(props);
for(int i = 0; i < 100; i++)
producer.send(new ProducerRecord<String, String>("my-topic", Integer.toString(i), Integer.toString(i)));
So, I guess, if you just write different topics in the ProducerRecord
, than it should be possible.
Also, here http://kafka.apache.org/081/documentation.html#producerapi it explicitly says that you can use a method send(List<KeyedMessage<K,V>> messages)
to write into multiple topics.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With