Containerized Kafka client errors when producing messages to the host Kafka server

Question

There are a number of similar types of queries on stackoverflow, but none quite match the problem that I am seeing.

I have a zookeeper/kafka setup on my server which work perfectly. One can produce

bin/kafka-console-producer.sh --broker-list 192.168.2.80:9092 --topic test

and consume

bin/kafka-console-consumer.sh --bootstrap-server 192.168.2.80:9092 --topic test --from-beginning

locally on the Linux Ubuntu 16.04 server.

From a Docker container - also running Ubuntu 16.04 - I want to produce and consume. The container's Kafka code was copied from that on the server. Firstly I can create a new topic

bin/kafka-topics.sh --create --zookeeper 192.168.2.80:2181 --replication-factor 1 --partitions 1 --topic test2

from the container and then list it again

bin/kafka-topics.sh --list --zookeeper 192.168.2.80:2181

However when I try to produce new messages, using the above (kafka-console-producer.sh) command it fails with the following message:

[2017-06-05 13:59:05,317] ERROR Error when sending message to topic test2  with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for test2-0: 1526 ms has passed since batch creation plus linger time

immediately after entering the text of the message and pressing enter.

It may seem strange running a Docker container on the same host, but once this works I will move the container to a separate host for production.

My kafka server.properties file:

listeners=PLAINTEXT://0.0.0.0:9092

Kafka version:

2.12-0.10.2.1

Docker version:

Docker version 1.12.6, build 78d1802

Answer 1

The problem is (slightly simplified) caused by how Kafka's protocol works. Given a list of "bootstrap servers" (e.g. localhost:9092), a Kafka client will contact those bootstrap servers, but then use the hostnames of the actual Kafka brokers as returned by the bootstrap servers (the broker's advertised.listeners config, depending on your Kafka/Docker setup, might be set to e.g. kafka:9092). So here, the client would talk to localhost:9092 for bootstrapping (which will work), but then switch to kafka:9092 (which will not work, "thanks" to the networking setup).

Fortunately there is a way to configure Kafka + Docker in a way that "just works", and it doesn't require shenanigans such as fiddling with your host's /etc/hosts file and such. As part of this you need to set a few (new) Kafka settings though, which were added in kafka's KIP-103: Separation of Internal and External traffic.

Here's a snippet for Docker Compose (docker-compose.yml) that demonstrates how to do this:

---
version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:3.2.1
    hostname: zookeeper
    ports:
      - '32181:32181'
    environment:
      ZOOKEEPER_CLIENT_PORT: 32181

  kafka:
    image: confluentinc/cp-kafka:3.2.1
    hostname: kafka
    ports:
      - '9092:9092'
      - '29092:29092'
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:32181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      # Following line is needed for Kafka versions 0.11+
      # in case you run less than 3 Kafka brokers in your
      # cluster because the broker config
      # `offsets.topic.replication.factor` (default: 3)
      # is now enforced upon topic creation
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Here, the key settings are:

• listener.security.protocol.map (which is being set via KAFKA_LISTENER_SECURITY_PROTOCOL_MAP)
• inter.broker.listener.name
• advertised.listeners

In the setup above, the containerized Kafka broker listens on localhost:9092 for access from your host machine (e.g. your Mac laptop) and on kafka:29092 for access from other containers.

A full end-to-end example is available at: https://github.com/confluentinc/cp-docker-images/blob/v3.2.1/examples/kafka-streams-examples/docker-compose.yml (documentation at http://docs.confluent.io/3.2.1/cp-docker-images/docs/tutorials/kafka-streams-examples.html).