Batch set data from Dictionary into Redis

Question

I am using StackExchange Redis DB to insert a dictionary of Key value pairs using Batch as below:

private static StackExchange.Redis.IDatabase _database;
public void SetAll<T>(Dictionary<string, T> data, int cacheTime)
{
    lock (_database)
    {
        TimeSpan expiration = new TimeSpan(0, cacheTime, 0);
        var list = new List<Task<bool>>();
        var batch = _database.CreateBatch();               
        foreach (var item in data)
        {
            string serializedObject = JsonConvert.SerializeObject(item.Value, Formatting.Indented,
        new JsonSerializerSettings { ContractResolver = new SerializeAllContractResolver(), ReferenceLoopHandling = ReferenceLoopHandling.Ignore });

            var task = batch.StringSetAsync(item.Key, serializedObject, expiration);
            list.Add(task);
            serializedObject = null;
        }
        batch.Execute();

        Task.WhenAll(list.ToArray());
    }
}

My problem: It takes around 7 seconds to set just 350 items of dictionary.

My question: Is this the right way to set bulk items into Redis or is there a quicker way to do this? Any help is appreciated. Thanks.

Answer 1

This second answer is kinda tangential, but based on the discussion it sounds as though the main cost is serialization:

The object in this context is big with huge infos in string props and many nested classes.

One thing you could do here is not store JSON. JSON is relatively large, and being text-based is relatively expensive to process both for serialization and deserialization. Unless you're using rejson, redis just treats your data as an opaque blob, so it doesn't care what the actual value is. As such, you can use more efficient formats.

I'm hugely biased, but we make use of protobuf-net in our redis storage. protobuf-net is optimized for:

• small output (dense binary without redundant information)
• fast binary processing (absurdly optimized with contextual IL emit, etc)
• good cross-platform support (it implements Google's "protobuf" wire format, which is available on just about every platform available)
• designed to work well with existing C# code, not just brand new types generated from a .proto schema

I suggest protobuf-net rather than Google's own C# protobuf library because of the last bullet point, meaning: you can use it with the data you already have.

To illustrate why, I'll use this image from https://aloiskraus.wordpress.com/2017/04/23/the-definitive-serialization-performance-guide/:

Notice in particular that the output size of protobuf-net is half that of Json.NET (reducing the bandwidth cost), and the serialization time is less than one fifth (reducing local CPU cost).

You would need to add some attributes to your model to help protobuf-net out (as per How to convert existing POCO classes in C# to google Protobuf standard POCO), but then this would be just:

using(var ms = new MemoryStream())
{
    foreach (var item in data)
    {
        ms.Position = 0;
        ms.SetLength(0); // erase existing data
        ProtoBuf.Serializer.Serialize(ms, item.Value);

        list.Add(_database.StringSetAsync(item.Key, ms.ToArray(), expiration));
    }
}

As you can see, the code change to your redis code is minimal. Obviously you would need to use Deserialize<T> when reading the data back.

---

If your data is text based, you might also consider running the serialization through GZipStream or DeflateStream; if your data is dominated by text, it will compress very well.