Redis aborting for OUT OF MEMORY

Question

I'm trying to move a big file (movie) into the redis cache in chunks. I'm using the stackexchange.redis on a Windows box. The redis is configured to do allkey-lru and to use append every second. Maxmemory is configured to 100mb.

ConnectionMultiplexer redis2 = ConnectionMultiplexer.Connect("localhost:6380, syncTimeout=100000");
IDatabase db2 = redis2.GetDatabase();

const int chunkSize = 4096;
string fileName = "D:\\movie.mp4";
using (var file = File.OpenRead(fileName))
{
    int bytesRead;
    int inCounter = 1;
    var buffer = new byte[chunkSize];
    while ((bytesRead = file.Read(buffer, 0, buffer.Length)) > 0)
    {
        db2.StringSet(file.Name + inCounter, buffer);
        inCounter++;
    }
}

When the chunk size is chunkSize = 4096 everything works great. However when I change the chunk size to 65536 the server crashes with the following log:

[2816] 20 Jul 23:06:42.300 * Starting automatic rewriting of AOF on 6766592700% growth
[2816] 20 Jul 23:06:42.331 * Background append only file rewriting started by pid 3672
[3672] 20 Jul 23:06:42.331 # Write error writing append only file on disk: Invalid argument
[3672] 20 Jul 23:06:42.331 # rewriteAppendOnlyFile failed in qfork: Invalid argument
[2816] 20 Jul 23:06:42.440 # fork operation complete
[2816] 20 Jul 23:06:42.440 # Background AOF rewrite terminated with error
[2816] 20 Jul 23:06:42.549 * Starting automatic rewriting of AOF on 7232582200% growth
[2816] 20 Jul 23:06:42.581 * Background append only file rewriting started by pid 1440
[2816] 20 Jul 23:06:42.581 # Out Of Memory allocating 10485768 bytes!
[2816] 20 Jul 23:06:42.581 #

=== REDIS BUG REPORT START: Cut & paste starting from here ===
[2816] 20 Jul 23:06:42.581 # ------------------------------------------------
[2816] 20 Jul 23:06:42.581 # !!! Software Failure. Press left mouse button to continue
[2816] 20 Jul 23:06:42.581 # Guru Meditation: "Redis aborting for OUT OF MEMORY" #..\src\redis.c:3467
[2816] 20 Jul 23:06:42.581 # ------------------------------------------------
[1440] 20 Jul 23:06:42.581 # Write error writing append only file on disk: Invalid argument
[1440] 20 Jul 23:06:42.581 # rewriteAppendOnlyFile failed in qfork: Invalid argument

Any ideas?

Answer 1

Turned out quite an interesting and a surprising problem!

The real cause of this is memory fragmentation in the allocator they are using (dlmalloc). Hopefully MSFT are going to make this better but I expect this would take some time.

In the meantime, the workaround.

Proper way of fixing this (for now)

Configure both maxmemory and maxheap parameters. Make maxheap much larger than maxmemory.

So if you want maxmemory=100MB then make maxheap 5x or even 10x larger, e.g maxheap=500MB or even maxheap=1000MB. I don't think there is a good rule of thumb how much larger maxheap needs to be which is why it's such a tricky problem.

The effect of this: Redis will still try and keep the memory usage under 100MB, but the actual physical memory in use may be larger than that, potentially up to maxheap value. How much exactly depends on the level of fragmentation. Hopefully in real life this would stay at reasonable levels.

---

I have logged the issue with the team. https://github.com/MSOpenTech/redis/issues/274

---

EDIT: I've completely reworked this answer in light of new knowledge. See previous versions in the edit history.