I work for a digital signage company where we send lots of animations/videos to various machines on the network. We have used ftp in the past to send files from a server app (which is our content management system) to machines that play the content.
Is it possible to use RabbitMQ in such an environment where we are sending large files (some videos may be a gigabyte or more in size)? Most files will be really small but occasionally we will send large videos.
Also some of the player machines will be added later which means that the queuing has to be dynamic (i.e. player added so now we can send content to it). Can RabbitMQ be configured in this way? (from the demos I have seen you have to create the queues up front before the server and client applications launch).
Also can we secure the sending of files so that the sending app has to log in before sending contents to a machine (like login for FTP)?
RabbitMQ allows multiple distributed RabbitMQ brokers to be connected in three different ways - with clustering, with federation and using shovel. We take advantage of these built-in plugins for multi-platform VOLTTRON communication.
While the theoretical message size limit in RabbitMQ is 2GB up to 3.7. 0, we don't recommend sending messages larger than 128MB, which is also the new max size limit in 3.8. 0 and onward. Large messages are especially problematic when using mirrored queues in HA clusters and can cause memory and performance issues.
The RabbitMQ message broker was deployed atop Google Compute Engine where it demonstrated the ability to receive and deliver more than one million messages per second (a sustained combined ingress/egress of over two million messages per second).
RabbitMQ has a plugin for consistent hash exchange. Using that exchange, and one consumer per queue, we can achieve message order with multiple consumers. The hash exchange distributes routing keys among queues, instead of messages among queues. This means all messages with the same routing key will go the same queue.
In order to add dynamic consumers of your files You can use EasyNetQ on top of your RabbitMQ broker with a publish/subscribe behavior.
As mentioned by Russ Cam, in the google group people have sent big messages but this could break a RabbitMQ cluster.
The solution could be an hybrid between FTP (or any other file transfer protocol) and Messages:
EasyNetQ already provides you a default subscription mechanism wich helps to subscribe dinamically (see link above) new consumers without configuring RabbitMQ in advance.
Finally you can think about splitting your files in many small message chunks but then you should create a function wich will reassemble the file on the receiver or worst you could lose some messages if your consumers start later.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With