Kafka works fine in the docker container. I can use docker exec -it [container name] [kafkascript]
and successfully create topics, produce/consume messages, but when I try from outside of the docker container using local kafka scripts I can only create and list topics. Producing and consuming messages throws errors:
Producing:
~/development/lib/kafka/kafka_2.11-0.10.0.0$ bin/kafka-console-producer.sh --broker-list $(docker-machine ip kafka):9092 --topic test
asdf
[2016-09-18 10:13:48,999] ERROR Error when sending message to topic test with key: null, value: 4 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Batch containing 1 record(s) expired due to timeout while requesting metadata from brokers for test-0
Consuming:
~/development/lib/kafka/kafka_2.11-0.10.0.0$ bin/kafka-console-consumer.sh --zookeeper $(docker-machine ip kafka):2181 --topic test --from-beginning
[2016-09-18 09:57:10,389] WARN Fetching topic metadata with correlation id 0 for topics [Set(test)] from broker [BrokerEndPoint(0,ba762186182f,9092)] failed (kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:110)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:80)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79)
at kafka.producer.SyncProducer.send(SyncProducer.scala:124)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2016-09-18 09:57:10,392] WARN [console-consumer-34526_3c15c2c24040-1474210630122-9404562b-leader-finder-thread], Failed to find leader for Set([test,0]) (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
kafka.common.KafkaException: fetching topic metadata for topics [Set(test)] from broker [ArrayBuffer(BrokerEndPoint(0,ba762186182f,9092))] failed
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:73)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
at kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
Caused by: java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:110)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:80)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:79)
at kafka.producer.SyncProducer.send(SyncProducer.scala:124)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
... 3 more
I'm using spotify/docker-kafka ,but I upgraded it to 0.10.0.0 and used some suggestions from jshark that sets up advertised.listeners. I'm running on a Mac. I've created a docker-machine called kafka. Here is my docker run:
docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`docker-machine ip kafka` --env ADVERTISED_PORT=9092 kafka
Here is my dockerfile:
# Kafka and Zookeeper
FROM java:openjdk-8-jre
ENV DEBIAN_FRONTEND noninteractive
ENV SCALA_VERSION 2.11
ENV KAFKA_VERSION 0.10.0.0
ENV KAFKA_HOME /opt/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION"
# Install Kafka, Zookeeper and other needed things
RUN apt-get update && \
apt-get install -y zookeeper wget supervisor dnsutils && \
rm -rf /var/lib/apt/lists/* && \
apt-get clean && \
wget -q http://apache.mirrors.spacedump.net/kafka/"$KAFKA_VERSION"/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz -O /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz && \
tar xfz /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz -C /opt && \
rm /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz
ADD scripts/start-kafka.sh /usr/bin/start-kafka.sh
# Supervisor config
ADD supervisor/kafka.conf supervisor/zookeeper.conf /etc/supervisor/conf.d/
# 2181 is zookeeper, 9092 is kafka
EXPOSE 2181
EXPOSE 9092
CMD ["supervisord", "-n"]
scripts/start-kafka.sh
For initial connections, Kafka clients need a bootstrap server list where we specify the addresses of the brokers. The list should contain at least one valid address to a random broker in the cluster. The client will use that address to connect to the broker.
Using Docker container networking, a Apache Kafka server running inside a container can easily be accessed by your application containers. Containers attached to the same network can communicate with each other using the container name as the hostname.
To start an Apache Kafka server, we'd first need to start a Zookeeper server. We can configure this dependency in a docker-compose. yml file, which will ensure that the Zookeeper server always starts before the Kafka server and stops after it.
This worked for me: https://stackoverflow.com/a/37655203/1839580
My summary: The ADVERTISED_HOST
environment variable in the spotify/kafka
container needs to change depending on whether your service is operating inside or outside the container. I an using Docker for Mac and I have my docker network set to bridged. Outside of Docker the ADVERTISED_HOST
needed to be set to localhost, inside of docker, it was set to myproject_kafka_1
or whatever it ends up being on your system. To fix it, I added and entry in my MacOS host files that mapped 127.0.0.1
to myproject_kafka_1
. I don't like messing with my host file, but it fixed this issue for me.
127.0.0.1 localhost
127.0.0.1 myproject_kafka_1
I wasn't able to interact with Kafka from outside its container until I added the following entries to server.properties:
listener.security.protocol.map=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
advertised.listeners=INSIDE://${container_ip}:9092,OUTSIDE://${outside_host_ip}:29092
listeners=INSIDE://:9092,OUTSIDE://:29092
inter.broker.listener.name=INSIDE
I posted a more complete answer here. Hope others won't spend as much time on this as I did.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With