using mqtt protocol with kafka as a message broker

Question

How can we use mqtt protocol with kafka as a message broker?

The clients(android/ios/desktop java apps etc) will be producing and consuming messages using mqtt phao client side libraries which are available in different languages using kafka as a message broker.

Any advice?

Answer 1

You can use a Kafka source connector which will stream data from an MQTT broker like Mosquitto into a Kafka cluster. See https://github.com/evokly/kafka-connect-mqtt

The simplest way to run the connector is in standalone mode, where a single instance will be running on the Kafka cluster on a single node. You can also run it in distributed mode (albeit with much more configuration) and this will distribute the connector across the cluster for greater throughput. In the distributed mode you can devise a topology that allows horizontal scaling, parallel throughput and high availability. Implementing additional guarantees requires additional load balancers, multiple MQTT brokers and last will and testament scenarios to deal with connectors crashing, but this is probably out of scope for this question.

Using the connector approach has the advantage of the Kafka cluster making sure your connector is alive and re-starting it if necessary. Distributed mode offers even more advantages.

Answer 2

This is not a good idea. A MQTT client usually is very light-weight with limited resources. Those devices or IoT has small memory/CPU power. A Kafka client generally is heavy-weight. For example, Kafka client has to keep track of the offset. It also requires interaction to Zookeepers. In short, Kafka is not suitable as MQTT broker. You are better off choosing a popular MQTT broker such as Mosquito.