Cannot Connect from Kafkacat running in docker to Kafka broker running locally on windows machine

Question

I am running kafka locally on a windows machine. The way I am running kafka is using

.\bin\windows\kafka-server-start.bat .\config\server.properties

The server.properties are:

listeners=Listener_BOB://:29092,Listener_Kafkacat://127.0.0.1:9092
advertised.listeners=Listener_BOB://:29092,Listener_Kafkacat://127.0.0.1:9092
listener.security.protocol.map=Listener_BOB:PLAINTEXT,Listener_Kafkacat:PLAINTEXT
inter.broker.listener.name=Listener_BOB

I am running kafkacat using docker

docker run -it --network="host" --name="producer" confluentinc/cp-kafkacat bash

When I run

 kafkacat -b host.docker.internal:9092 -C -t test

I get the error message:

% ERROR: Local: Broker transport failure: 127.0.0.1:9092/0: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT)

I understand that I can run Kafka in docker but I would like to know why I cannot connect to the broker and produce or consume messages. I tried different things and tried to understand what the listeners do but I couldn't wrap my head around why this doesn't work.

When I do

kafkacat -b host.docker.internal:9092 -t test -L

I get

1 brokers:
  broker 0 at 127.0.0.1:9092
 1 topics:
  topic "test" with 1 partitions:
    partition 0, leader 0, replicas: 0, isrs: 0

Maybe someone can explain what am I doing wrong or tell me why this is not possible.

I downloaded the latest version of Kafka: kafka-2.4 Machine is Windows 10 Docker for Windows running using Linux containers

Answer 1

You need to set host.docker.internal:9092 as an advertised listener. Not localhost / 127.0.0.1

That's the address that clients will be returned when trying to connect

When you do that, you should be able to see this

1 brokers:
  broker 0 at host.docker.internal:9092

Otherwise, as you can tell, the bootstrap connection worked, but the advertised address is going to be 127.0.0.1 to reach that specific broker, which obviously doesn't work because that would be the kafkacat container itself

Note: --network="host" doesn't do what you expect outside of a Linux host machine, so you're best off removing it