Bidirectional messaging system using kafka

Question

Is there any possible to develop a bi-directional messaging system using apache kafka ? I need to subscribe for a topic from my consumer as well as I need to send message from my consumer.

Answer 1

You could do it one of two ways. Either set up a prefix system for the message keys or put content inside of the message that allows the consumer to avoid messages it has produced.

Now as to whether you should design it like this, that depends on your message traffic. If you're not slamming it with events, it might be better to consider something like Thrift as a way to have your message components do bidirectional communication. Where Kafka really excels relative to its complexity is when you need to produce and consume massive volumes of data. That might not be the case for you.

For example, one common use case with Kafka is to hook it up to a service like Storm, Apex or Samza for doing distributed processing of hundreds of GB or even TB of data. If your system has a high throughput requirement, that architecture would be a good one to consider as a starting point with Kafka for handling messages. With Storm, if you need to send messages back for reprocessing, you can always use the Kafka bolt to republish a message into Kafka to ensure it gets completely reprocessed.