How to implement request-reply (synchronous) messaging paradigm in Kafka?

Question

I am going to use Kafka as a message broker in my application. This application is written entirely using Python. For a part of this application (Login and Authentication), I need to implement a request-reply messaging system. In other words, the producer needs to get the response of the produced message from the consumer, synchronously. Is it feasible using Kafka and its Python libraries (kafka-python, ...) ?

Answer 1

I'm facing the same issue (request-reply for an HTTP hit in my case) My first bet was (100% python):

1. start a consumer thread,
2. publish the request message (including a request_id)
3. join the consumer thread
4. get the answer from the consumer thread The consumer thread subscribe to the reply topic (seeked to end) and deals with received messages until finding the request_id (modulus timeout)

If it works for a basic testing, unfortunatly, creating a KafkaConsumer object is a slow process (~300ms) so it's not an option for a system with massive traffic. In addition, if your system deals with parallel request-reply (for example, multi-threaded like a web server is) you'll need to create a KafkaConsumer dedicated to request_id (basically by using request_id as consumer_group) to avoid to have reply to request published by thread-A consumed (and ignored) by thread-B. So you can't here reclycle your KafkaConsumer and have to pay the creation time for each request (in addition to processing time on backend). If your request-reply processing is not parallelizable you can try to keep the KafkaConsuser object available for threads started to get answer

The only solution I can see at this point is to use a DB (relational/noSQL):

1. requestor store request_id in DB (as local as possible) aznd publish request in kafka
2. requestor poll DB until finding answer to request_id In parallel, a consumer process receiving messages from reply topic and storing result in DB

But I don't like polling..... It wil generate heavy load on DB in a massive traffic system

My 2CTS