Exception running kafka-console-producer.sh (0.8.1.1)

Question

I am trying out the kafka-console-producer.sh. I type something to the console and hit enter and I get tons of stack traces to the console like this. Appreciate any pointers.

The command to launch the producer is:

./bin/kafka-console-producer.sh --broker-list localhost:2181 --topic test

Then I type some string and hit enter and see the following in the console.

[2014-11-24 15:27:36,022] ERROR fetching topic metadata for topics [Set(test)] from broker [ArrayBuffer(id:0,host:localhost,port:2181)] failed (kafka.utils.Utils$)
kafka.common.KafkaException: fetching topic metadata for topics [Set(test)] from broker [ArrayBuffer(id:0,host:localhost,port:2181)] failed
    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:67)
    at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
    at kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:78)
    at kafka.utils.Utils$.swallow(Utils.scala:167)
    at kafka.utils.Logging$class.swallowError(Logging.scala:106)
    at kafka.utils.Utils$.swallowError(Utils.scala:46)
    at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78)
    at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
    at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
    at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
    at scala.collection.immutable.Stream.foreach(Stream.scala:547)
    at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
    at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
Caused by: java.io.EOFException: Received -1 when reading from channel, socket has likely been closed.
    at kafka.utils.Utils$.read(Utils.scala:376)
    at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
    at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
    at kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
    at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
    at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:74)
    at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
    at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
    ... 12 more

I see a warning on the zookeeper console as well:

WARN Exception causing close of session 0x0 due to java.io.EOFException (org.apache.zookeeper.server.NIOServerCnxn)    

I used the following to setup my topic:

./bin/kafka-topics.sh --create --zookeeper localhost:2181 --partitions 1 --replication-factor 1 --topic test

and I am able to list my topic:

./bin/kafka-topics.sh --list --zookeeper localhost:2181

Answer 1

Your broker-list argument is pointing to Zookeeper instead of the actual broker. The proper usage would be:

./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

assuming your broker runs on port 9092 (default).

You can refer here for more information (your issue is described in Step 4: Send some messages)

Answer 2

This is because you are starting your consumer in a wrong way. If you will check the official documentation, you can see that you have to:

start Zookeper

bin/zookeeper-server-start.sh config/zookeeper.properties

start Kafka

bin/kafka-server-start.sh config/server.properties

create a topic

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

start producing messages for a topic

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test 

Notice the difference: you are connecting to zookeper (port 2181) you need to connect to a broker (port 9092)