Sending data with kafka-python only working when briefly delaying code

Question

I'm sending some data to a Kafka topic using kafka-python. I struggled with not being able to send data to my Kafka topic for a while until I found out that if I delay the code briefly it works.

from kafka import KafkaProducer
from time import sleep

producer = KafkaProducer(bootstrap_servers="localhost:9092")
producer.send("topic", "foo")
sleep(.1)

This code does not work for me without using sleep(.1). It's like sending data needs time to settle for it to work properly. Is there anything in the kafka-python client that deals with this? Or a better solution?

Answer 1

A year later, but to anyone seeing this, a solution is below. The issue here is race condition with the end of the script and the send call, which is why the sleep() command works.

The kafka module should better handle the python exit, or at the minimum output something to standard out/error, so this behavior isn't silent.

From the kafka-python github:

# Block until a single message is sent (or timeout)
future = producer.send('foobar', b'another_message')
result = future.get(timeout=60)

Now you can guarantee that your script will block until a message has been confirmed published.