I am having trouble with KafaConsumer
to make it read from the beginning, or from any other explicit offset.
Running the command line tools for the consumer for the same topic , I do see messages with the --from-beginning
option and it hangs otherwise
$ ./kafka-console-consumer.sh --zookeeper {localhost:port} --topic {topic_name} --from-beginning
If I run it through python, it hangs, which I suspect to be caused by incorrect consumer configs
consumer = KafkaConsumer(topic_name,
bootstrap_servers=['localhost:9092'],
group_id=None,
auto_commit_enable=False,
auto_offset_reset='smallest')
print "Consuming messages from the given topic"
for message in consumer:
print "Message", message
if message is not None:
print message.offset, message.value
print "Quit"
Consuming messages from the given topic (hangs after that)
I am using kafka-python 0.9.5 and the broker runs kafka 8.2. Not sure what the exact problem is.
Set _group_id=None_ as suggested by dpkp to emulate the behavior of console consumer.
I ran into the same problem: I can recieve in kafka console but can't get message with python script using package kafka-python
.
Finally I figure the reason is that I didn't call producer.flush()
and producer.close()
in my producer.py
which is not mentioned in its documentation .
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With