How to write a file to Kafka Producer

Question

I am trying to load a simple text file instead of standard input in Kafka. After downloading Kafka, I performed the following steps:

Started zookeeper:

bin/zookeeper-server-start.sh config/zookeeper.properties

Started Server

bin/kafka-server-start.sh config/server.properties

Created a topic named "test":

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Ran the Producer:

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test  Test1 Test2 

Listened by the Consumer:

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning Test1 Test2 

Instead of Standard input, I want to pass a data file or even a simple text file to the Producer which can be seen directly by the Consumer. Any help would really be appreciated. Thanks!

Answer 1

You can pipe it in:

kafka-console-producer.sh --broker-list localhost:9092 --topic my_topic --new-producer < my_file.txt 

Found here.

From 0.9.0:

kafka-console-producer.sh --broker-list localhost:9092 --topic my_topic < my_file.txt