Accessing Local Kafka from within Services deployed in Local Docker For Mac (incl. Kubernetes extension)

Question

• I'm working locally on OSX.
  • I'm using Kafka and zookeeper in local mode. Meaning zookeeper from my Kafka installation. One node cluster.
  • Both work on the loopback localhost

zookeeper.connect=localhost:2181

• My /etc/host looks as follows:

127.0.0.1         localhost MaatPro.local  
255.255.255.255   broadcasthost  
::1               localhost MaatPro.local  
fe80::1%lo0       localhost MaatPro.local

• I have docker for Mac set on my machine, with the Kubernetes extension.

My scenario

• I have an Akka-stream micro-service dockerized, that reads data from an external database and write it in a Kafka topic. It uses as bootstrap server:

"localhost:9092"

Issue

• When I run my service on my machine directly (e.g. command line or from within Intellij) everything works fine. When I run it on my Local Docker or Kubernetes I get the following error:

(o.a.k.clients.NetworkClient) [Producer clientId=producer-1] Connection to node 0 could not be established. The broker may not be available.

With Kubernetes I build the following YAML File to deploy my pod:

apiVersion: v1
kind: Pod
metadata:
  name: fetcher
spec:
  hostNetwork: true
  containers:
   - image: entellectextractorsfetchers:0.1.0-SNAPSHOT
     name: fetcher

I took the precaution to set hostNetwork: true

With Docker daemon directly I originally tried to set the network as host too but discovered that this does not work with docker for mac. Hence, I abandoned that route. I understood that it has to do with virtualization.

1) Does the virtualization issue that happens with docker is actually the same as my local kubernetes? Basically, the host network is the virtual machine and not my mac?

2) I try to change my code and add as a bootstrap server the following address: host.docker.internal as per the documentation. But the problem persists. Is the fundamental problem the fact that I am working on a loopback address? Shall I work on my network address indeed? To which address does host.docker.internal point to? How can I make it work with the loopback address? If I'm completely off, any idea of what I need to implement to get this working?

Thank you so much for any help with this.

Answer 1

Based on @cricket_007 guidance Kafka Listeners - Explained and the many read I have had here and there over this issue, including the official docker documentation I Want to connect from a container to a servicer on the host

I came up with the following solution.

I added the following to my default local kafka configuration (i.e. server.properties)

listeners=EXTERNAL://0.0.0.0:19092,INTERNAL://0.0.0.0:9092
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
advertised.listeners=INTERNAL://127.0.0.1:9092,EXTERNAL://docker.for.mac.localhost:19092
inter.broker.listener.name=INTERNAL

In fact External here, is expected to be the docker network. This config is only for my OSX machine for my local development purpose. I do not expect people connecting to my laptop to use my local kafka hence i can use EXTERNAL://docker.for.mac.localhost:19092. This is what is advertised to my container in docker/kube. From within that network, docker.for.mac.localhost is reachable.

Note this would probably not work with Minikube. This is specific to Docker For Mac. The kubernetes that I run on my machine is the one coming with docker for mac and not minikube.

Finally in my code i use both in a list

"localhost:9092, docker.for.mac.localhost:19092"

I use typeSafe config, so that in prod, this is erased by the env variable. When the env variable is not specified, this is what is used. When i run my micro-service from Intellij localhost:9092 is used. That’s because In that case, i am in the same network as my kafka/zookeeper in my machine. However when I run the same micro-service from docker/kube docker.for.mac.localhost:19092 is used.

Answers to the side questions I had

1. Yes. Docker for Mac use HyperKit as a lightweight virtual machine, running a linux on it, and Docker Engine is ran on it. The Docker for MAC Kubernetes extension is basically about running kubernetes cluster Services/infrastructure as containers in the docker daemon. Docker for Mac vs. Docker Toolbox . In other words, the host is hyperkit and not osx. But as the above doc explain, Docker for Mac implementation is all about making it appear to the user as if there were no virtualization involve between OSX and Docker.

2. • Connecting to the host using loopback address is an issue that has not been solved. I'm not even sure that it works perfect even if the host is Linux. Not sure, might have been resolved at this point. Nonetheless, it would require to run an image by stating that the container or the pod in the case of kube are on the host network. But in docker for mac, that functionality will never work based on my readings online. Hence the solution of using docker.for.mac.localhost or host.docker.internal, that Docker for Mac did set up to refer to the mac host and not the hyperkit host.

   • host.docker.internal and docker.for.mac.localhost are one of the same and the late recommendation at this point is host.docker.internal. This being said, this address did not originally work for me because my Kafka Set up was not good. It is worth readying @criket_007 link to understand that well http://rmoff.net/2018/08/02/kafka-listeners-explained.

Answer 2

Following @MaatDeamon approach I only did the following and it worked for me.

advertised.listeners=PLAINTEXT://:9092

And in your application config properties

localhost:9092,host.docker.internal:9092