Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Kafka Docker - Can't produce or consume from outside of docker container

Kafka works fine in the docker container. I can use docker exec -it [container name] [kafkascript] and successfully create topics, produce/consume messages, but when I try from outside of the docker container using local kafka scripts I can only create and list topics. Producing and consuming messages throws errors:

Producing:

~/development/lib/kafka/kafka_2.11-0.10.0.0$ bin/kafka-console-producer.sh --broker-list $(docker-machine ip kafka):9092 --topic test asdf

[2016-09-18 10:13:48,999] ERROR Error when sending message to topic test with key: null, value: 4 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Batch containing 1 record(s) expired due to timeout while requesting metadata from brokers for test-0

Consuming:

~/development/lib/kafka/kafka_2.11-0.10.0.0$ bin/kafka-console-consumer.sh --zookeeper $(docker-machine ip kafka):2181 --topic test --from-beginning

[2016-09-18 09:57:10,389] WARN Fetching topic metadata with correlation id 0 for topics [Set(test)] from broker [BrokerEndPoint(0,ba762186182f,9092)] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
    at kafka.network.BlockingChannel.send(BlockingChannel.scala:110)
    at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:80)
    at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79)
    at kafka.producer.SyncProducer.send(SyncProducer.scala:124)
    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
    at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2016-09-18 09:57:10,392] WARN [console-consumer-34526_3c15c2c24040-1474210630122-9404562b-leader-finder-thread], Failed to find leader for Set([test,0]) (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
kafka.common.KafkaException: fetching topic metadata for topics [Set(test)] from broker [ArrayBuffer(BrokerEndPoint(0,ba762186182f,9092))] failed
    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:73)
    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
    at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
Caused by: java.nio.channels.ClosedChannelException
    at kafka.network.BlockingChannel.send(BlockingChannel.scala:110)
    at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:80)
    at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79)
    at kafka.producer.SyncProducer.send(SyncProducer.scala:124)
    at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
    ... 3 more

I'm using spotify/docker-kafka ,but I upgraded it to 0.10.0.0 and used some suggestions from jshark that sets up advertised.listeners. I'm running on a Mac. I've created a docker-machine called kafka. Here is my docker run:

 docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`docker-machine ip kafka` --env ADVERTISED_PORT=9092 kafka

Here is my dockerfile:

# Kafka and Zookeeper

FROM java:openjdk-8-jre

ENV DEBIAN_FRONTEND noninteractive
ENV SCALA_VERSION 2.11
ENV KAFKA_VERSION 0.10.0.0
ENV KAFKA_HOME /opt/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION"

# Install Kafka, Zookeeper and other needed things
RUN apt-get update && \
    apt-get install -y zookeeper wget supervisor dnsutils && \
    rm -rf /var/lib/apt/lists/* && \
    apt-get clean && \
    wget -q http://apache.mirrors.spacedump.net/kafka/"$KAFKA_VERSION"/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz -O /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz && \
    tar xfz /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz -C /opt && \
    rm /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz

ADD scripts/start-kafka.sh /usr/bin/start-kafka.sh

# Supervisor config
ADD supervisor/kafka.conf supervisor/zookeeper.conf /etc/supervisor/conf.d/

# 2181 is zookeeper, 9092 is kafka
EXPOSE 2181
EXPOSE 9092

CMD ["supervisord", "-n"]

scripts/start-kafka.sh

like image 952
Jonathan Avatar asked Sep 18 '16 15:09

Jonathan


People also ask

How do I connect to a Docker container in Kafka?

For initial connections, Kafka clients need a bootstrap server list where we specify the addresses of the brokers. The list should contain at least one valid address to a random broker in the cluster. The client will use that address to connect to the broker.

Can Kafka run on Docker?

Using Docker container networking, a Apache Kafka server running inside a container can easily be accessed by your application containers. Containers attached to the same network can communicate with each other using the container name as the hostname.

How do I use Kafka and Zookeeper Docker?

To start an Apache Kafka server, we'd first need to start a Zookeeper server. We can configure this dependency in a docker-compose. yml file, which will ensure that the Zookeeper server always starts before the Kafka server and stops after it.


2 Answers

This worked for me: https://stackoverflow.com/a/37655203/1839580

My summary: The ADVERTISED_HOST environment variable in the spotify/kafka container needs to change depending on whether your service is operating inside or outside the container. I an using Docker for Mac and I have my docker network set to bridged. Outside of Docker the ADVERTISED_HOST needed to be set to localhost, inside of docker, it was set to myproject_kafka_1 or whatever it ends up being on your system. To fix it, I added and entry in my MacOS host files that mapped 127.0.0.1 to myproject_kafka_1. I don't like messing with my host file, but it fixed this issue for me.

127.0.0.1   localhost
127.0.0.1   myproject_kafka_1
like image 56
Wil Avatar answered Sep 17 '22 23:09

Wil


I wasn't able to interact with Kafka from outside its container until I added the following entries to server.properties:

listener.security.protocol.map=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT

advertised.listeners=INSIDE://${container_ip}:9092,OUTSIDE://${outside_host_ip}:29092

listeners=INSIDE://:9092,OUTSIDE://:29092

inter.broker.listener.name=INSIDE

I posted a more complete answer here. Hope others won't spend as much time on this as I did.

like image 23
João Matos Avatar answered Sep 20 '22 23:09

João Matos