Interact with kafka docker container from outside of docker host [duplicate]

Question

I have built a kafka docker container and orchestrate it using docker-compose.

Calling docker ps I get the following putput:

CONTAINER ID        IMAGE                          COMMAND                CREATED             STATUS              PORTS                                         NAMES
    5bde6f76246e        hieutrtr/docker-kafka:0.0.1    "/start.sh"            About an hour ago   Up About an hour    7203/tcp, 0.0.0.0:32884->9092/tcp             dockerkafka_kafka_3
    be354f1b8cc0        hieutrtr/docker-ubuntu:devel   "/usr/bin/supervisor   About an hour ago   Up About an hour    22/tcp                                        producer1
    50d3203af90e        hieutrtr/docker-kafka:0.0.1    "/start.sh"            About an hour ago   Up About an hour    7203/tcp, 0.0.0.0:32883->9092/tcp             dockerkafka_kafka_2
    61b285f39615        hieutrtr/docker-kafka:0.0.1    "/start.sh"            2 hours ago         Up 2 hours          7203/tcp, 0.0.0.0:32882->9092/tcp             dockerkafka_kafka_1
    20c9c5ccec05        jplock/zookeeper:3.4.6         "/opt/zookeeper/bin/   2 hours ago         Up 2 hours          2888/tcp, 3888/tcp, 0.0.0.0:32881->2181/tcp   dockerkafka_zookeeper_1

I can run a producer and a consumer from inside the docker container, but it is not working from outside the docker network.

For example:

I run a kafka producer on my localhost and the following error appears:

$ kafka_2.9.1-0.8.2.1: bin/kafka-console-producer.sh --topic test --broker-list $DOCKER_HOST:32884
[2015-08-31 06:55:15,450] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
to
[2015-08-31 06:55:20,214] WARN Failed to send producer request with correlation id 2 to broker 1 with data for partitions [test,0] (kafka.producer.async.DefaultEventHandler)
java.nio.channels.ClosedChannelException
    at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
    at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
    at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
    at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:103)
    at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:103)
    at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:103)

This is my kafka docker example on github that includes the mentioned problem.

So, is anyone experiencing the same problems and can help me in any way?

Additional info:

(Just fork from ches/kafka and modify something for docker-compose) :

Answer 1

In the Kafka server properties you need to set advertised.listeners to the ip/port of your running container and then it should work.

Answer 2

Here are my two cents, since I had a hard time figuring this one out.

My $KAFKA_HOME/config/server.properties contains the following:

listener.security.protocol.map=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT

advertised.listeners=INSIDE://${container_ip}:9092,OUTSIDE://${outside_host_ip}:29092

listeners=INSIDE://:9092,OUTSIDE://:29092

inter.broker.listener.name=INSIDE

This is creating two connections, one to be used inside docker and another to be used outside. You have to choose a new port for the latter, in my case 29092, make sure this port is exposed and mapped by docker.

I was not yet able to figure out a solution without the ${outside_host_ip} in the environment, hence I'm providing the host machine's ip as an env var.

Test:

1. Enter the Kafka container and create a topic: ./kafka-topics.sh -zookeeper zookeeper:2181 --create --topic dummytopic --partitions 1 --replication-factor 1
2. From outside the Kafka container do: ./kafka-console-producer.sh --broker-list 0.0.0.0:29092 --topic dummytopic and input a message

I hope this helps others