redis structure, performance

Question

I am newbie to redis, and I have a dataset of several million member IDs, emails and usernames, and am thinking about storing them for example in list structures. I think list and sorted set may be best suitable for my case.

Right now, I am using the first letter of the username to index into a list and push data to the back list: rpush list:name:a username,member_id. However, since the list is not sorted, will retrieving a certain record within several millions of entries be slow?

Will a sorted set (because it is sorted) be better than a list in this case? Or, do you have any other recommendation to increase performance?

The key to access records should be username and email.

Answer 1

Accessing a list by any index that isn't near the front or end will be expensive, costing O(N). For large lists, this is not very efficient.

Using hashes may be a better fit for your needs. This will use more memory than a list, but will provide nearly O(1) access.

A hash in redis is a named key that can contain arbitrary fields and values.

You can store the entire user record in a single redis hash, named using the member_id (hopefully this is a short value). If the member_id is guaranteed to be unique per-user, here is how to populate a hash for user with member_id 42.

hset user:42 email [email protected]
hset user:42 username foobar
hset user:42 logincount 0

The redis "key name" here is "user:42". Each user will get a single key, similar to a single row in a SQL database, but more flexible. You can then update two auxiliary hashes: one to map usernames to member_id, and another to map email addresses to member_id. This assumes you have a 1:1 relationship among member_id, username and email address.

hset username_to_id foobar 42
hset email_to_id [email protected] 42

When you need to look up the email address for a particular user, you first look up the member_id from the email_to_id hash and then retrieve the email field from the hash at key user:member_id Likewise, you can start with a username, look up the member_id in the username_to_id hash, and then get to the user record stored in the user:member_id hash.

Here is an example for looking up the username given an email address:

redis> hget email_to_id [email protected]
"42"
redis> hget user:42 username
"foobar"
redis> 

You can add more records to the user by adding more fields to the "user:" hash. If you want to increment a login counter, that is straightforward as well:

redis> hincrby user:42 login_count 1
(integer) 1
redis> hgetall user:42
1. "email"
2. "[email protected]"
3. "username"
4. "foobar"
5. "login_count"
6. "1"
redis> 

You can find more information about hashes on the redis.io site.