Redis: Excluding values from sorted set based on hash field value

Question

I was wondering if anyone could provide some suggestions on how to make sorted set generation more efficient?

I am working on a project where ranking data is calculated on an hourly basis and stored in a database. The data can be filtered by member gender, country, etc. There is roughly 2 million rows that need to be processed and it takes a long time.

We want to move to a more real time approach where data is stored / sorted / filtered in Redis and a daily clean rebuild.

In my prototype, I am creating a sorted set for each possible combination of filters e.g.: leaderboard.au.male, leaderboard.au.female, etc. I've scripted this process but once you handle every case it means there is 118 sorted sets created.

Ideally, I'd like to have a single ranking sorted set and hash sets for each member containing their name, gender and country. Then using Redis only return sorted set values based on the user defined filters. (e.g. only get rankings for males from Australia).

Is this possible to do natively in Redis?

Answer 1

I suggest you keep a set with the rankings for all members:

leaderboard = { id1: score1, id2: score2, ... }

And a set for each type (gender, country etc):

members.male = { id1, id2, ... }
members.au = { id2, id3, ... }

Then, you do a ZINTERSTORE:

zinterstore leaderboard.male 2 leaderboard members.male

Or, to get a leaderboard of male AU members:

zinterstore leaderboard.au.male 3 leaderboard members.male members.au

You can control how the scoring for the resulting sorted set should be calculated using WEIGHTS and AGGREGATE.

If you don't want to keep the resulting sets for long, you could then EXPIRE them, and only build a new set if it doesn't exist.