Issue in connecting kafka from outside

Question

I am using hortonwork Sandbox for kafka server trying to connect kafka from eclipse with java code . Use this configuration to connect to producer to send the message

metadata.broker.list=sandbox.hortonworks.com:45000
serializer.class=kafka.serializer.DefaultEncoder
zk.connect=sandbox.hortonworks.com:2181
request.required.acks=0
producer.type=sync

where sandbox.hortonworks.com is sandboxname to whom i connect

in kafka server.properties I changed this configuration

host.name=sandbox.hortonworks.com

advertised.host.name=System IP(on which my eclipse is running)
advertised.port=45000

did the port forwarding also ,

I am able to connect to kafka server from eclipse but while sending the message get the exception Exception"Failed to send messages after 3 tries."

Answer 1

First make sure you have configured host-only network for your Hortonworks Sandbox VM as described here:

http://hortonworks.com/community/forums/topic/use-host-only-networking-for-the-virtual-machine/

After doing this your sandbox VM should get a IP (e.g. 192.168.56.101) and it should be reachable from your host via SSH like

$ ssh [email protected]

Then open Ambari at http://192.168.56.101:8080/ and change the Kafka configuration to

listeners=PLAINTEXT://0.0.0.0:6667
advertised.listeners=PLAINTEXT://192.168.56.101:6667 

The latter property must be added in the section "Custom kafka-broker" (See also http://hortonworks.com/community/forums/topic/ambari-alerts-how-to-change-kafka-port/).

Then start/restart Kafka via Ambari. You should now be able to access Kafka from outside the Hortonworks Sandbox VM. You can test this (from outside of the sandbox VM) using e.g. the Kafka console producer from the Kafka distribution like

$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
$ bin/kafka-console-producer.sh --topic test --broker-list 192.168.56.101:6667

Answer 2

After almost one week of config tweeking, I finally get it work. Similar to asmaier's answer but if you are using cloud server like me: Azure Sandbox-hdp, try to pub/sub thru remote consumer/producer.

In azure:

first SSH into your azure in Ambari Web-UI localhost:8080, add

listeners=PLAINTEXT://0.0.0.0:6667
advertised.listeners=PLAINTEXT://127.0.0.1:6667

in terminal @root, Set up docker port forwarding like hortonwroks instruction page sandbox

vi start_scripts/start_sandbox.sh

add port 6667 on the list

On your PC:

first SSH into your azure plus tunneling 6667.

then write in cmd: or your run own java/c# script

kafka\bin\windows>kafka-console-producer --broker-list localhost:9092 --topic test

Only thing that bothers me right now is I cant find a way to make kafka push messages directly to a real public IP/ Azure. It seems Data traffic thru broker can only operate inside the docker internally.