Menu
I'm currently migrating some data to Redis and I'm considering using a sorted set to store approximately 1.4e6 items (with associated scores/counts). Is this number of items in a set likely to exceed a practical limit, making it too painful to use the set? I plan on running 64 bit redis, so available memory for the data should not be a problem. Does anyone have experience with a sorted set this size? If so, how are your insertion and query times for the set?
305
Introduction. Redis is an open-source, in-memory key-value data store. In Redis, sorted sets are a data type similar to sets in that both are non repeating groups of strings. The difference is that each member of a sorted set is associated with a score, allowing them to be sorted from the smallest score to the largest.
Redis lists are lists of strings sorted by insertion order.
Redis ZADD command adds all the specified members with the specified scores to the sorted set stored at the key. It is possible to specify multiple score/member pairs.
Redis ZRANGE command returns the specified range of elements in the sorted set stored at the key. The elements are considered to be ordered from the lowest to the highest score. Lexicographical order is used for elements with an equal score.
It depends what you want to do with the set. The simple operations are mostly O(log n) which means that they take only twice as long for a million item set as they do for a thousand item set. Unless you have something seriously broken in your config like a memory limit smaller than the set, performance shouldn't be a problem.
Where you need to be careful is with operations on multiple sets, particularly union - that will take a thousand times longer for the million item set. In practical terms this isn't necessarily a problem though - either it will be fast enough for your purposes anyway (redis has commands documented as too slow for production use that are still best measured in milliseconds) or you can adjust the order of operations to avoid running union on really large sets.
100
Our site has a sorted set with about 2 milions items (email addresses) with integer scores and it took up about 320MB in memory size.
30
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
