Menu
I'm trying to start a kafka service using docker-compose, and it should be able to be accessed inside and outside docker. So, it should be matter of setting the right advertisers inside and outside:
version: '3' services: zookeeper: image: wurstmeister/zookeeper ports: - "2181:2181" kafka: image: wurstmeister/kafka ports: - "9094:9092" environment: KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://127.0.0.1:9094 KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://127.0.0.1:9094 KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE The problem is that when I try to connect from outside the cluster, I don't get 127.0.0.1 as the name of the node, but the internal hostname:
$ kafkacat -L -b 127.0.0.1:9094 Metadata for all topics (from broker -1: 127.0.0.1:9092/bootstrap): 1 brokers: broker 1001 at 91588ea968d4:9092 28 topics: ... Isn't the purpose of KAFKA_ADVERTISED_LISTENERS and KAFKA_LISTENERS to handle that situation? I tried setting KAFKA_ADVERTISED_HOST_NAME but it's ignored (one piece of documentation says it's deprecated, other one says that it's still active), but nevertheless that doesn't seem to be the answer, since I want two different advertised hostnames for two different networks.
I guess the old question remains: how to make kafka work inside and outside docker-compose?
289
listeners configuration parameter in the server properties file ( <path-to-confluent>/etc/kafka/server. properties ). KAFKA_LISTENER_SECURITY_PROTOCOL_MAP. Defines key/value pairs for the security protocol to use, per listener name. This is equivalent to the listener.
To make a port available to services outside of Docker, or to Docker containers which are not connected to the container's network, use the --publish or -p flag. This creates a firewall rule which maps a container port to a port on the Docker host to the outside world.
To start an Apache Kafka server, we'd first need to start a Zookeeper server. We can configure this dependency in a docker-compose. yml file, which will ensure that the Zookeeper server always starts before the Kafka server and stops after it.
What I needed to do was to declare the LISTENERS as both binding to all interfaces, and then advertise them differently - one to the docker network, one to the host:
services: zookeeper: image: confluentinc/cp-zookeeper ports: - "2181:2181" environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 ZOOKEEPER_SYNC_LIMIT: 2 kafka: image: confluentinc/cp-kafka ports: - 9094:9094 depends_on: - zookeeper environment: KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_LISTENERS: INTERNAL://0.0.0.0:9092,OUTSIDE://0.0.0.0:9094 KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,OUTSIDE://localhost:9094 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
