Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can I use RabbitMQ to distribute large files to multiple machines?

I work for a digital signage company where we send lots of animations/videos to various machines on the network. We have used ftp in the past to send files from a server app (which is our content management system) to machines that play the content.

Is it possible to use RabbitMQ in such an environment where we are sending large files (some videos may be a gigabyte or more in size)? Most files will be really small but occasionally we will send large videos.

Also some of the player machines will be added later which means that the queuing has to be dynamic (i.e. player added so now we can send content to it). Can RabbitMQ be configured in this way? (from the demos I have seen you have to create the queues up front before the server and client applications launch).

Also can we secure the sending of files so that the sending app has to log in before sending contents to a machine (like login for FTP)?

like image 918
JD. Avatar asked Jan 28 '14 23:01

JD.


People also ask

Can RabbitMQ be distributed?

RabbitMQ allows multiple distributed RabbitMQ brokers to be connected in three different ways - with clustering, with federation and using shovel. We take advantage of these built-in plugins for multi-platform VOLTTRON communication.

How much data can RabbitMQ handle?

While the theoretical message size limit in RabbitMQ is 2GB up to 3.7. 0, we don't recommend sending messages larger than 128MB, which is also the new max size limit in 3.8. 0 and onward. Large messages are especially problematic when using mirrored queues in HA clusters and can cause memory and performance issues.

How many messages can RabbitMQ handle per second?

The RabbitMQ message broker was deployed atop Google Compute Engine where it demonstrated the ability to receive and deliver more than one million messages per second (a sustained combined ingress/egress of over two million messages per second).

Can RabbitMQ have multiple consumers?

RabbitMQ has a plugin for consistent hash exchange. Using that exchange, and one consumer per queue, we can achieve message order with multiple consumers. The hash exchange distributes routing keys among queues, instead of messages among queues. This means all messages with the same routing key will go the same queue.


1 Answers

In order to add dynamic consumers of your files You can use EasyNetQ on top of your RabbitMQ broker with a publish/subscribe behavior.

As mentioned by Russ Cam, in the google group people have sent big messages but this could break a RabbitMQ cluster.

The solution could be an hybrid between FTP (or any other file transfer protocol) and Messages:

  • Every time your client want to transfer a file it publish a message to the bus with a reference to the file (e.g. the FTP url)
  • The consumers of this message will then connect through FTP in order to download the file

EasyNetQ already provides you a default subscription mechanism wich helps to subscribe dinamically (see link above) new consumers without configuring RabbitMQ in advance.

Finally you can think about splitting your files in many small message chunks but then you should create a function wich will reassemble the file on the receiver or worst you could lose some messages if your consumers start later.

like image 93
Ganto Avatar answered Nov 09 '22 15:11

Ganto