Menu
I had a question regarding Kafka broker clusters on AWS. Right now there is an AWS ELB sitting in front of the cluster, but when I set the "bootstrap.servers" property of my producer or consumer to the "A" record (and correct port number) of my ELB, both the producer and consumer fail to produce and consume messages respectively. I have turned off all SSL on my broker and am connecting through the PLAINTEXT 9092 port, with which my ELB forwards port 1234 to 9092. So in my Producer Configs properties for example, I will have...
bootstrap.servers = ("A" record of ELB):1234
More info:
Has anyone had any luck running Kafka behind an ELB? If so, please help me out!
403
What is Kafka loading balancing? Load balancing with Kafka is a straightforward process and is handled by the Kafka producers by default. While it isn't traditional load balancing, it does spread out the message load between partitions while preserving message ordering.
Usually, GraphQL is used in the frontend with a server implemented in Node. js, while Kafka is often used as an integration layer between backend components.
A Kafka broker receives messages from producers and stores them on disk keyed by unique offset. A Kafka broker allows consumers to fetch messages by topic, partition and offset. Kafka brokers can create a Kafka cluster by sharing information between each other directly or indirectly using Zookeeper.
There is only 1 controller at a time.
You can use an ELB as the bootstrap.servers, but the brokers still need to be directly accessible to the client. The ELB will be used for the initial metadata request the client makes to figure out which topic partitions are on which brokers, but after that it'll use the hostname of the server (or advertised.listeners setting if you need to customize it, which, e.g. might be necessary on EC2 instances to get the public IP of a server).
58
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
