Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Redis vs Kafka vs RabbitMQ for 1MB messages

I am currently researching a queueing solution to handle medium sized messages of 1MB. Besides the features differences between Redis, Kafka and RabbitMQ I cannot find any good answer to their performance on messages of size around 1MB.

  1. Any of you guys knows how many messages of 1MB can any of these handle?
  2. Do you know any other queueing solutions which can perform better?
like image 623
Slava Shpitalny Avatar asked Jul 18 '17 08:07

Slava Shpitalny


People also ask

Is Redis faster than RabbitMQ?

RabbitMQ is designed as a dedicated message broker, whereas Redis is a database that you can use as a message broker. Mostly, RabbitMQ outperforms Redis and guarantees message delivery with the help of message durability and acknowledgments.

Is Kafka faster than Redis?

As mentioned above, Redis is an in-memory store. This means that it uses its primary memory for storage and processing which makes it much faster than the disk-based Kafka.

Is Redis a good message broker?

Redis Pub/Sub is an extremely lightweight messaging protocol designed for broadcasting live notifications within a system. It's ideal for propagating short-lived messages when low latency and huge throughput are critical. Redis Lists and Redis Sorted Sets are the basis for implementing message queues.

Is RabbitMQ faster than Kafka?

Kafka offers much higher performance than message brokers like RabbitMQ. It uses sequential disk I/O to boost performance, making it a suitable option for implementing queues. It can achieve high throughput (millions of messages per second) with limited resources, a necessity for big data use cases.


2 Answers

When you are evaluating Kafka vs Redis in your case, there are other factors which you have to take into account, besides message size. Here are some of them I can think of:

  • How many producers/consumers? Redis performance can be affected in case of greater number of producers/consumers due to the nature of Redis (push based queue). This is because Redis delivers the message to all the consumers at once, at the moment the message is put in the queue.
  • Do you need speed or reliability first? If speed is of utmost importance, use Redis since it does not persist messages and it will deliver them faster. If you need reliability use Kafka since it persist messages even after they are delivered.
  • Do you want your consumers to get messages once they are ready or you want messages to be sent to the consumers immediately? In first case use Kafka because it's pull based mechanism (consumer have to ask for the message). In second case use Redis since it's push based mechanism (message is pushed to the consumer once it's on the queue). RabbitMQ is also push based (although there is pull API with bad performance)
  • What is the number of messages expected? If it's not huge use Redis since you are limited with memory. Otherwise use Kafka. Best practice for RabbitMQ is to keep queues short. This means that you can consume messages at the close rate at which they appear on the queue. So if you have some long lasting operation on the consumer part probably RabbitMQ is not the best choice.
  • Scaling? Kafka scales horizontally really well (it's built with scalability in mind). RabbitMQ is usually scaled vertically. Redis also scales well horizontally if needed.

It's obvious that there are more than one criteria when you evaluate proper queueing solution. There are best practices and recommendations for each of the queueing engines that you are looking at. Think more about your specific use case, it's definitely worth the time since it will save you time later on if you chose inappropriate queueing engine.

like image 173
buhtla Avatar answered Oct 22 '22 03:10

buhtla


I am answering for Kafka.

Kafka itself has very good performance even for big messages. In our tests with 2 Kafka nodes we reach p2p communication with 170 MB/sec smaller messages 150 MB/s bigger messages.

The only thing you need to remember is to configure the broker to accept bigger messages.

Hier is nice article: Configuring Kafka for Performance and Resource Management - Handling Large Messages

I know other p2p solution which might be interesting when you have concrete requirements look at YAMI4

I was using Redis but only for very small messages, so I cannot say anything about 1MB.

like image 24
Seweryn Habdank-Wojewódzki Avatar answered Oct 22 '22 03:10

Seweryn Habdank-Wojewódzki