Which MQTT server for > 1M connections [closed]

Question

With a budget of perhaps a few million to setup a MQTT server farm how would you do so?

It must have the following properties:

• Support for 4-5M connections across all data centers.
• 300k msg/s of around 1kb each
• Geographic redundancy
• Can't lose messages QOS1
• Each client will publish to a single topic but subscribe to their own unique topic. This implies 4-5 million topics.

MQTT Server brokers can be found here: https://github.com/mqtt/mqtt.github.io/wiki/server-support#capabilities

However capabilities are usually not published.

Answer 1

Although the Erlang powered VerneMQ MQTT broker is still quite new, there is nothing (besides RAM/CPU/IPs/Bandwidth) that should prevent you from opening that many connections.

http://verne.mq

make sure to set something similar to:

listener.max_connections = infinity
listener.nr_of_acceptors = 1000
erlang.max_ports = 10000000
erlang.process_limit = 10000000

in your vernemq.conf

disclaimer: I'am one of the devs of VerneMQ and happy to help you reach your 1M connections/server.

Answer 2

HiveMQ is a self hosted, java based Enterprise MQTT broker, specifically designed to support millions of concurrent connections.

The HiveMQ team has done a benchmark, connecting more than 10.000.000 concurrent MQTT clients to a HiveMQ broker cluster. To reach this number, accompanied with a decent performance, some configuration needed to be done on the operating system of the used machines.

Open files in /etc/security/limits.conf:

hivemq  hard    nofile  1000000
hivemq  soft    nofile  1000000
root    hard    nofile  1000000
root    soft    nofile  1000000

TCP tweaking in /etc/sysctl.conf

# This causes the kernel to actively send RST packets when a service is overloaded.
net.ipv4.tcp_fin_timeout = 30

# The maximum file handles that can be allocated.
fs.file-max = 5097152

# Enable fast recycling of waiting sockets.
net.ipv4.tcp_tw_recycle = 1

# Allow to reuse waiting sockets for new connections when it is safe from protocol viewpoint.
net.ipv4.tcp_tw_reuse = 1

# The default size of receive buffers used by sockets.
net.core.rmem_default = 524288

# The default size of send buffers used by sockets.
net.core.wmem_default = 524288

# The maximum size of received buffers used by sockets.
net.core.rmem_max = 67108864

# The maximum size of sent buffers used by sockets.
net.core.wmem_max = 67108864

# The size of the receive buffer for each TCP connection. (min, default, max)
net.ipv4.tcp_rmem = 4096 87380 16777216

# The size of the sent buffer for each TCP connection. (min, default, max)
net.ipv4.tcp_wmem = 4096 65536 16777216

Details on the VMs used, the specific configurations needed on the OS side and detailed performance results can all be found in the 10 Million Benchmark Paper

Disclaimer: I am part of the HiveMQ Team.