Load blancing MQTT broker

Question

Is it possible to load balance MQTT broker? Can I use ELB to load balance MQTT? Any pointers in this direction would be helpful. I hit upon http://www.slideshare.net/kellogh/mqtt-kafka-33100776 but it did not help much.

Answer 1

Load balancing MQTT brokers is not supported out-of-the-box by most MQTT brokers. There are brokers available which support clustering, though. Take a look at this list to find out a broker which suits your clustering requirements.

This blog post is a bit dated but might still be useful if you want to learn about MQTT broker clustering: http://www.hivemq.com/building-a-high-availability-mqtt-cluster/

To answer your question: The MQTT broker HiveMQ works perfectly with AWS ELB for load balancing across several availability zones. You might want to increase the default ELB timeout for TCP connections to your MQTT keep-alive time you are using for your application. You can even use Auto Scaling groups for scaling elastically if needed. HiveMQ can use S3 for the discovery of other cluster nodes so you can add and remove cluster nodes at runtime.

I don't know if other brokers work as seamlessly with the AWS ELB. If HiveMQ does not suit your needs, best would be to search for a broker in the list linked above and try it :)

Disclaimer: I'm working for the company which develops HiveMQ.

Answer 2

Use HAProxy. A free, fast and reliable solution offering high availability, load balancing and proxying for TCP and HTTP based applications. It’s suited for very high traffic web sites.

Answer 3

You can't load balance MQTT Brokers like you would load balance web servers! The LB solution has to persist the connection on the MQTT Topic field, NOT the client's IP:Port...otherwise not all subscribers will get published messages. A 'Publisher' message only gets sent to a single MQTT Broker, so only that Broker will resend the messages. If a 'Subscriber' for the same MQTT Topic goes to a different Broker, it won't be receiving the messages from the Publisher connected to the other Broker. You can, of course, bridge your MQTT Brokers so that all of them get the message, but then what's the point of LB-ing them?? You're not really scaling them at that point, which is probably why you want to LB them in the first place. An F5 BIG-IP can be used to do true MQTT LB, IF you create the iRule script to do the LB decision on the MQTT Topic. There are companies out there doing this today.

Answer 4

(Adding to @JD Allens response) Load balancing MQTT or TCP in general will mostly not work as intended if you keep state on servers like MQTT(subscriptions). For example if you have two MQTT backend servers behind the LB, half of your MQTT clients will connect to one MQTT broker and the other half to the other. Since these two brokers are not interconnected by any means, client will have no info about of what is going on in the other broker (subscriptions and publishes). You have two options:

• Just configure the load balancer that will direct all connections to a single broker and only fail over to the other in case of error. Only one broker should be up at a given time (this is not load balancing per se).

• Use a clustered MQTT broker like HiveMQ, EMQX etc. They will interconnect the backend brokers.