Searching in values of a redis db

Question

I am a novice in using Redis DB. After reading some of the documentation and looking into some of the examples on the Internet and also scanning stackoverflow.com, I can see that Redis is very fast, scales well but this costs the price that we have to think out how our data will be accessed at the design time and what operations they will have to undergo. This I can understand but I am a little confused about searching in the data what was so easy, however slow, with the plain old SQL. I could do it in one way with the KEY command but it is an O(N) operation and not O(log(N)). So I would lose one of the advantages of Redis.

What do more experienced colleagues say here?

Let's take an example use case: we have need to store personal data for approx. 100.000 people and those data need to be searched by name, phone nr.

For this I would use the following structures:

1. SET for storing all persons' ids {id1, id2, ...}  2. HASH for each person to store personal data and name it  like map:<id> e.g. map:id1{name:<name>, phone:<number>, etc...} 

Solution 1:

1. HASH for storing all persons' ids but the key should be the phone number 2. Then with the command KEY 123* all ids could be retrieved who have a phone number  sarting with 123. On basis of the ids also the other personal data could be retrieved. 3. So forth for each data to be searched for a separate HASH should be created. 

But a major drawback of this solution is that the attribute values must also be unique, so that the assigment of the phone number and the ids in the HASH would be unambiguous. On the other hand, O(N) runtime is not ideal.

Moreover, this uses more space than would be necessary and the KEY command deteriorates the access performance. (http://redis.io/commands/keys)

How should it be done in the right way? I could also imagine that ids would go in a ZSET and the data needed search could be the scores but this make only possible to work with ranges not with seraches.

Thank you also in advance, regards, Tamas

Answer summary: Actually, both responses state that Redis was not designed to search in the values of the keys. If this use case is necessary, then either workarounds need to be implemented as shown in my original solution or in the below solution.

The below solution by Eli has a much better performance, than my original one because the access to the keys can be considered constant, only the list of ids needs to be iterated through, for the access this would give O(const) runtime. This data model also allows that one person might have the same phone number as someone else and so on also for names etc... so 1-n relationship is also possible (I would say with old ERD terminology).

The drawback of this solution is, that it consumes much more space than mine and phone numbers whose starting digits are known only, could not be searched.

Thanks for both responses.

Answer 1

Redis is for use cases where you need to access and update data at very high frequency and where you benefit from use of data structures (hashes, sets, lists, strings, or sorted sets). It's made to fill very specific use cases. If you have a general use case like very flexible searching, you'd be much better served by something built for this purpose like elastic search or SOLR.

That said, if you must do this in Redis, here's how I'd do it (assuming users can share names and phone numbers):

name:some_name -> set([id1, id2, etc...]) name:some_other_name -> set([id3, id4, etc...])  phone:some_phone -> set([id1, id3, etc...]) phone:some_other_phone -> set([id2, id4, etc...])  id1 -> {'name' : 'bob', 'phone' : '123-456-7891', etc...} id2 -> {'name' : 'alice', 'phone' : '987-456-7891', etc...} 

In this case, we're making a new key for every name (prefixed with "name:") and every phone number (prefixed "phone:"). Each key points to a set of ids that have all the info you want for a user. When you search, for a phone, for example, you'll do:

HGETALL 'phone:123-456-7891' 

and then loop through the results and return whatever info on each (name in our example) in your language of choice (you can do this whole thing in server-side Lua on the Redis box to go even faster and avoid network back-and-forth, if you want):

for id in results:     HGET id 'name' 

You're cost here will be O(m) where m is the number of users with the given phone number, and this will be a very fast operation on Redis because of how optimized it is for speed. It'll be overkill in your case because you probably don't need things to go so fast, and you'd prefer having flexible search, but this is how you would do it.

Answer 2

redis is awesome, but it's not built for searching on anything other than keys. You simply cant query on values without building extra data sets to store items to facilitate such querying, but even then you don't get true search, just more maintenance, inefficient use of memory, yada, yada...

This question has already been addressed, you've got some reading to do :-D

To search strings, build auto-complete in redis and other cool things...
How do I search strings in redis?

Why using MongoDB over redis is smart when searching inside documents... What's the most efficient document-oriented database engine to store thousands of medium sized documents?