StackExchange.Redis Scan x amount of keys

Question

I have a redis db that has thousands of keys and I'm currently running the following line to get all the keys:

string[] keysArr = keys.Select(key => (string)key).ToArray();

But because I have a lot of keys this takes a long time. I want to limit the number of keys being read. So I'm trying to run an execute command where I get 100 keys at a time:

var keys = Redis.Connection.GetDatabase(dbNum).Execute("scan", 0, "count", 100);

This command successfully runs the command, however unable to access the the value as it is private, and unable to cast it even though RedisResult classs provides a explicit cast to it:

public static explicit operator string[] (RedisResult result);

Any ideas to get x amount of keys at a time from redis?

Thanks

Answer 1

SE.Redis has a .Keys() method on IServer API which fully encapsulates the semantics of SCAN. If possible, just use this method, and consume the data 100 at a time. It is usually pretty easy to write a batching function, i.e.

ExecuteInBatches(server.Keys(), 100, batch => DoSomething(batch));

with:

public void ExecuteInBatches<T>(IEnumerable<T> source, int batchSize,
        Action<List<T>> action)
{
    List<T> batch = new List<T>();
    foreach(var item in source) {
        batch.Add(item);
        if(batch.Count == batchSize) {
             action(batch);
             batch = new List<T>(); // in case the callback stores it
        }
    }
    if (batch.Count != 0) {
        action(batch); // any leftovers
    }
}

The enumerator will worry about advancing the cursor.

---

You can use Execute, but: that is a lot of work! Also, SCAN makes no gaurantees about how many will be returned per page; it can be zero - it can be 3 times what you asked for. It is ... guidance only.

Incidentally, the reason that the cast fails is because SCAN doesn't return a string[] - it returns an array of two items, the first of which is the "next" cursor, the second is the keys. So maybe:

var arr = (RedisResult[])server.Execute("scan", 0);
var nextCursor = (int)arr[0];
var keys = (RedisKey[])arr[1];

But all this is doing is re-implementing IServer.Keys, the hard way (and significantly less efficiently - ServerResult is not the ideal way to store data, it is simply necessary in the case of Execute and ScriptEvaluate).