Kafka with Docker dynamic advertised_host_name

Question

I've been using wurstmeister/Kafka for a few weeks now in Dev and QA, but in each case I need to hard-code KAFKA_ADVERTISED_HOST_NAME to the IP of the box that it's on, using docker-compose. This hasn't been a problem during testing, but now that I'm trying to scale this out to production, it's becoming a little bit more frustrating.

I'm continuing to use docker-compose to somewhat manually deploy three instances of Kafka and Zookeper onto three separate cloud hosts. I've opened up the appropriate ports, and attempted everything in my limited Docker knowledge to dynamically assign KAFKA_ADVERTISED_HOST_NAME. Much to my dismay, it always yields some sort of error. The README on docker hub mentions assigning this variable dynamically VIA
HOSTNAME_COMMAND, e.g. HOSTNAME_COMMAND: "route -n | awk '/UG[ \t]/{print $$2}'"

This causes my application to obtain a connection refused response when attempting to connect. However, manually assigning the IP to the three hosts works perfectly fine. What am I missing here?!

Answer 1

Compose can substitute variables into configuration options at run time.

Compose Environment variables

Set the KAFKA_ADVERTISED_HOST_NAME container environment variable to a local variable called DOCKER_HOST_IP.

whatever:
  environment:
    KAFKA_ADVERTISED_HOST_NAME: ${DOCKER_HOST_IP}

Then DOCKER_HOST_IP needs to be set whenever you run docker-compose. You will get a warning from docker-compose when it's not set.

IP on the Docker host

Running ip route show will list the default interface.
Then ip address show will give you the ip addresses.

To get these into a variable

default_interface=$(ip ro sh | awk '/^default/{ print $5; exit }')
export DOCKER_HOST_IP=$(ip ad sh $default_interface | awk '/inet /{print $2}' | cut -d/ -f1)
[ -z "$DOCKER_HOST_IP" ] && (echo "No docker host ip address">&2; exit 1 )
echo "$DOCKER_HOST_IP"

You can add those commands to whatever your startup script is, or create a standalone script from them to call when you need it.

IP via Docker Machine

If you are managing a remote docker-machine you can get the ip via the machine environment.

DOCKER_HOST_IP=$(docker-machine ip ${DOCKER_MACHINE_NAME})