Menu
I need to store huge amount of binary files (10 - 20 TB, each file ranging from 512 kb to 100 MB).
I need to know if Redis will be efficient for my system. I need following properties in my system:
I intend to use a cluster of commodity hardware to reduce costing as much as possible. Please suggest pros and cons of building such a system using Redis. I am also concerned about high ram requirements of Redis.
387
Large amount of data: Redis does not fit as a Database if we need to store very large data sets, or expect our data to grow very fast.
In Redis, we have the advantage of storing both strings and collections of strings as values. A string value cannot exceed 512 MB of text or binary data. However, it can store any type of data, like text, integers, floats, videos, images, and audio files.
You can store up to 512 megabytes in a Redis string. It can store any type of data, like text, integers, floats, videos, images, or audio files. In this example, SET and GET are Redis commands, which we will discuss later. name is the key, and educative is the string value that we are storing.
You can store the binary file in filesystem and then store the path to the file in redis. This is the recommended method if you want to manage binary data with redis. You can also store binary files/data directly inside redis. Redis keys and values don't have to be text strings.
I would not use Redis for such a task. Other products will be a better fit IMO.
Redis is an in-memory data store. If you want to store 10-20 TB of data, you will need 10-20 TB of RAM, which is expensive. Furthermore, the memory allocator is optimized for small objects, not big ones. You would probably have to cut your files in various small pieces, it would not be really convenient.
Redis does not provide an ad-hoc solution for HA and failover. A master/slave replication is provided (and works quite well), but with no support for the automation of this failover. Clients have to be smart enough to switch to the correct server. Something on server-side (but this is unspecified) has to switch the roles between master and slaves nodes in a reliable way. In other words, Redis only provides a do-it-yourself HA/failover solution.
Sharding has to be implemented on client-side (like with memcached). Some clients have support for it, but not all of them. The fastest client (hiredis) does not. Anyway, things like rebalancing has to be implemented on top of Redis. Redis Cluster which is supposed to support such sharding capabilities is not ready yet.
I would suggest to use some other solutions. MongoDB with GridFS can be a possibility. Hadoop with HDFS is another one. If you like cutting edge projects, you may want to give the Elliptics Network a try.
163
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
