Menu
KAFKA_ADVERTISED_HOST_NAME or KAFKA_ADVERTISED_LISTENERS in Kafka RC YML that ultimately set ADVERTISED_HOST_NAME/ADVERTISED_LISTENERS properties in server.properties either does not help accessing kafka from outside of kubernetes cluster. Please suggest how can I publish/consume from outside of kubernetes cluster. Thanks much!
812
You can expose the Kafka cluster outside the Kubernetes cluster by declaring one or more externalListeners in the KafkaCluster custom resource. Above, externalListeners creates two external access points through which the Kafka cluster's brokers can be reached. These external listeners are registered in the advertized.
Pod/Container Port Not Exposed Use kubectl exec -it POD_NAME /bin/SHELL (where shell is often bash if it is installed in the container, or sh otherwise) to get a shell running inside the pod. The shell actually runs inside one of its containers.
Ways to connect You have several options for connecting to nodes, pods and services from outside the cluster: Access services through public IPs. Use a service with type NodePort or LoadBalancer to make the service reachable outside the cluster. See the services and kubectl expose documentation.
I had the same problem with accessing kafka from outside of k8s cluster on AWS. I manage to solve this issue by using kafka listeners feature which from version 0.10.2 supports multiple interfaces.
here is how I configured kafka container.
ports:
- containerPort: 9092
- containerPort: 9093
env:
- name: KAFKA_ZOOKEEPER_CONNECT
value: "zookeeper:2181"
- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
value: "INTERNAL_PLAINTEXT:PLAINTEXT,EXTERNAL_PLAINTEXT:PLAINTEXT"
- name: KAFKA_ADVERTISED_LISTENERS
value: "INTERNAL_PLAINTEXT://kafka-internal-service:9092,EXTERNAL_PLAINTEXT://123.us-east-2.elb.amazonaws.com:9093"
- name: KAFKA_LISTENERS
value: "INTERNAL_PLAINTEXT://0.0.0.0:9092,EXTERNAL_PLAINTEXT://0.0.0.0:9093"
- name: KAFKA_INTER_BROKER_LISTENER_NAME
value: "INTERNAL_PLAINTEXT"
Apart from that I configured two Services. One for internal(Headless) & one for external(LoadBalancer) communication.
Hopefully this will save people's time.
127
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
