Menu
I was exploring the fact whether Kafka supports priority for any queue or message to process.
It seems it doesn't support any such thing. I googled and found this mail archive which supports this also: http://mail-archives.apache.org/mod_mbox/incubator-kafka-users/201206.mbox/%3CCAOeJiJhVHsr=d6aSTihPsqWVg6vK5xYLam6yMDcd6UAUoXf-DQ@mail.gmail.com%3E
Does anyone here configured of Kafka to prioritize any topic or message?
306
Ideally, we should separate messages by priority using different consumer groups. Messages with higher priority would fall into one group while messages with less priority would fall into another group, and then each group could have a different number of consumers to work on messages.
In Kafka, order can only be guaranteed within a partition. This means that if messages were sent from the producer in a specific order, the broker will write them to a partition and all consumers will read from that in the same order.
We can use Kafka as a Message Queue or a Messaging System but as a distributed streaming platform Kafka has several other usages for stream processing or storing data. We can use Apache Kafka as: Messaging System: a highly scalable, fault-tolerant and distributed Publish/Subscribe messaging system.
In queue, you only have one receiver or consumer; unlike in topic where in you can have your message be disseminated to a number of subscribers. Also, in topic, the publisher has to be continuously active for a subscriber to receive the messages. Otherwise the message will be reallocated.
Kafka is a fast, scalable, distributed in nature by its design, partitioned and replicated commit log service.So there is no priority on topic or message.
I also faced same problem that you have.Solution is very simple.Create topics in kafka queue,Let say:
high_priority_queue
medium_priority_queue
low_priority_queue
Publish high priority message in high_priority_queue and medium priority message in medium_priority_queue.
Now you can create kafka consumer and open stream for all topic.
// this is scala code
val props = new Properties()
props.put("group.id", groupId)
props.put("zookeeper.connect", zookeeperConnect)
val config = new ConsumerConfig(props)
val connector = Consumer.create(config)
val topicWithStreamCount = Map(
"high_priority_queue" -> 1,
"medium_priority_queue" -> 1,
"low_priority_queue" -> 1
)
val streamsMap = connector.createMessageStreams(topicWithStreamCount)
You get stream of each topic.Now you can first read high_priority topic if topic does not have any message then fallback on medium_priority_queue topic. if medium_priority_queue is empty then read low_priority queue.
This trick is working fine for me.May be helpful for you!!.
105
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
