Menu
We are trying to consume from a Kafka Cluster using the Java Client. The Cluster is a behind a Jump host and hence the only way to access is through a SSH Tunnel. But we are not able read because once the consumer fetches metadata it uses the original hosts to connect to brokers. Can this behaviour be overridden? Can we ask Kafka Client to not use the metadata?
458
The Kafka distribution provides a command utility to see messages from the command line. It displays the messages in various modes. Kafka provides the utility kafka-console-consumer.sh which is located at ~/kafka-training/kafka/bin/kafka-console-producer.sh to receive messages from a topic on the command line.
You can expose the Kafka cluster outside the Kubernetes cluster by declaring one or more externalListeners in the KafkaCluster custom resource. Above, externalListeners creates two external access points through which the Kafka cluster's brokers can be reached. These external listeners are registered in the advertized.
Not as far as I know.
The trick I used when I needed to do something similar was:
/etc/hosts file so that the advertised hostname of broker n is resolved to the ip of the virtual interface n.Es.
Kafka brokers:
Virtual interfaces:
Tunnels:
ssh -L 192.168.1.1:9092:broker1.mykafkacluster:9092 jumphost ssh -L 192.168.1.2:9092:broker1.mykafkacluster:9092 jumphost /etc/hosts:
If you configure your system like this you should be able reach all the brokers in your Kafka cluster.
Note: if you configured your Kafka brokers to advertise an ip address instead of a hostname the procedure can still work but you need to configure the virtual interfaces with the same ip address that the broker advertises.
199
You don't actually have to add virtual interfaces to acces the brokers via SSH tunnel if they advertise a hostname. It's enough to add a hosts entry in /etc/hosts of your client and bind the tunnel to the added name.
Assuming broker.kafkacluster is the advertised.hostname of your broker:
/etc/hosts:
127.0.2.1 broker.kafkacluster
Tunnel:ssh -L broker.kafkacluster:9092:broker.kafkacluster:9092 <brokerhostip/name>
33
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
