Redis cache design

Question

I want to integrate a caching layer in my node.js API. I have never built one before so I have a few questions.

I have objects called "containers"

I want to look up these containers by id. Often multiple containers at once. The lookups are not consistent and every user will have a different set of lookup ids.

I do not need to query the data at the present time. So I started out by just using a key/value store where the key was something like "container_1", with the data being a serialized json representation.

But I have to look up multiple containers at once efficiently. I noticed the hash data type so now I do" hmset containers [id] [serialized json]. This way I can do hmget containers 1 3 4 to return containers 1,3,4.

Would it be better to store the hashes in redis as real objects like hmset containers:1 name test-container? Is this efficient or a normal way of handling the data? How does this strategy scale to tens or hundreds of thousands of entries from the perspective of time complexity? Can I use the expire key on sets?

Thanks

Answer 1

There are several questions here. I will do my best to answer all of them.

It sounds like you are proposing three possible storage scenarios. Here are some notes on the consequences of each.

Option #1: Store each of your serialized containers in a string key

You can retrieve multiple containers at once easily using MGET. This option should be equal in performance to storing all of the containers in a single hash. This option will require slightly more memory, as top-level keys have more overhead than hash fields. You get the advantages of using a top-level key, so you can expire containers individually and use other key commands like DUMP/RESTORE/OBJECT/MIGRATE on individual containers.

Options #2: Store all of your serialized containers in a single hash, each one in a separate field in the hash

As you mentioned, HMGET will let you retrieve multiple containers at once. This option is slightly more memory efficient than option #1. This also allows your top-level key space to remain small since it doesn't grow with every container. This advantage isn't significant, but is a small administrative aid since you can use the KEYS command with less pain. It should be just as fast as option #1.

Option #3: Store each of your containers in a hash, in a somewhat unserialized format, with the hash fields corresponding to the container properties

If each container is a JSON object this should map pretty well. You will still have to decide what to do when the values of the object are not simple strings. You may still have to store each value as JSON or some other serialized format. This solution is arguably the most complicated. There may be a performance hit when trying to re-create the original container in javascript as each property will need to be parsed independently and reconstituted into a final object unless the driver is doing this for you somewhere.

This approach may make it easier and more performant to retrieve specific fields of a container.

Retrieving multiple containers in one command would be more difficult with this approach as it would require pipelining or Lua scripting.

Conclusion

The appropriateness, scalability, and time complexity of each approach depends greatly on your access pattern. Are you attempting to search by the value of container properties? Then, option #3 starts looking attractive. Otherwise, options #1 and #2 look the most attractive. Ideally you will use additional keys to build indexes of your data for various use cases. You might have a set with the ids of containers belonging to a user, or a list with the container ids ordered by last updated time.

All of these approaches are reasonable and using additional indexes can help ensure any of them could be made to scale.

Can I use the expire key on sets?

Yes. You can expire any key, regardless of the type. Strings, hashes, lists, sets, etc.