Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Kafka : How to connect kafka-console-consumer to fetch remote broker topic content?

I have setup a kafka zookeeper and 3 brokers on one machine on ec2 with ports 9092..9094 and am trying to consume the topic content from another machine. The ports 2181 (zk), 9092, 9093 and 9094 (servers) are open to the consumer machine. I can even do a bin/kafka-topics.sh --describe --zookeeper 172.X.X.X:2181 --topic remotetopic which gives me

Topic:remotetopic PartitionCount:1 ReplicationFactor:3 Configs: Topic: remotetopic Partition: 0 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1 Blockquote

However when i do bin/kafka-console-consumer.sh --zookeeper 172.X.X.X:2181 --from-beginning --topic remotetopic I get

WARN Fetching topic metadata with correlation id 0 for topics [Set(remotetopic)] from broker [id:0,host:localhost,port:9092] failed (kafka.client.ClientUtils$) java.nio.channels.ClosedChannelException

Why is the consumer trying to read from localhost? Is there any option or commandline or default file where this is read from; and I can change it?

Any help would be appreciated!

like image 496
notrai Avatar asked Nov 07 '15 14:11

notrai


People also ask

How does Kafka console consumer connect to Kafka cluster?

Step 1: Start the zookeeper as well as the kafka server initially. Step2: Type the command: 'kafka-console-consumer' on the command line. This will help the user to read the data from the Kafka topic and output it to the standard outputs.

Does Kafka client connect to all brokers?

A client that wants to send or receive messages from the Kafka cluster may connect to any broker in the cluster. Every broker in the cluster has metadata about all the other brokers and will help the client connect to them as well, and therefore any broker in the cluster is also called a bootstrap server.

Which command is used in Kafka to retrieve messages from a topic?

Kafka provides the utility kafka-console-consumer.sh which is located at ~/kafka-training/kafka/bin/kafka-console-producer.sh to receive messages from a topic on the command line.


2 Answers

I have created a python script which allows you to access kafka/zookeeper via ssh tunnel to consume and produce messages from your local machine.

Compatible for Linux & Mac

Usage: kafkatunnel.py [OPTIONS] COMMAND [ARGS]...

  Access kafka/zookeeper via ssh tunnel to consume and produce messages from
  your local machine

Commands:
  aws     retrieve kafka/zookeeper ip's from AWS...
  manual  provide the IP's of your zookeeper/kafka

try it: https://github.com/simple-machines/kafka-tunnel

like image 157
Felix Avatar answered Sep 21 '22 18:09

Felix


I had same problem , I am using kafka 0.11 and I started producer and broker in my Ubuntu VM and consumer in Windows 10.

I had changed (and uncommented)

advertised.listeners=PLAINTEXT://your.host.name:9092

to

advertised.listeners=PLAINTEXT://<myActual IP address>:9092

example:

advertised.listeners=PLAINTEXT://192.168.150.150:9092

from file config/server.properties at server/broker side (i.e. in my case it was Ubuntu VM)

like image 24
nomadSK25 Avatar answered Sep 22 '22 18:09

nomadSK25