IOREDIS - Error Trying to Migrate from Redis to KeyDB

Question

We were using Redis for a plenty of time until we have come to the conclusion that moving to KeyDB may be a good choice for its features.

Environment

OS: Centos 7
NodeJs: v12.18.0
Redis: v6.0.5
Targeted KeyDB: v0.0.0 (git:1069d0b4) //  keydb-cli -v showed this. Installed Using Docker.
ioredis: v4.17.3
pm2: v4.2.1 // used for clustering my application. 

Background

Referring to the KeyDB documentation, KeyDB is compatible with the latest version of Redis.

KeyDB remains fully compatible with Redis modules API and protocol. As such a migration from Redis to KeyDB is very simple and would be similar to what you would expect migrating in a Redis to Redis scenario. https://docs.keydb.dev/docs/migration/

In the same page they provide a list of redis clients which are compatible with KeyDB. The list contains ioredis which I'm using.

KeyDB is compatible with all Redis clients as listed here so this should not be a concern. Simply use your client as you would with Redis. https://docs.keydb.dev/docs/migration/

Problem

As said in the documentation. I should be able to migrate easily to KeyDB in a few hours. Well that is not the case! At least not for me! I spent my last 3 days searching on the internet for the solution. I came to the conclusion that I should write to stackoverflow :)

The issue is somehow interesting. The Client is actually working with KeyDb and the process is actually setting and retrieving keys (Not sure but may lose some data during the error.). But On 10% of time it gives me the following error, And continues to work again after a while. As I'm using Redis for storing sessions and other stuff on my production environment; I can not risk ignoring such insisting error.

error:  message=write EPIPE, stack=Error: write EPIPE
./app-error-1.log:37:    at WriteWrap.onWriteComplete [as oncomplete] (internal/stream_base_commons.js:92:16), errno=EPIPE, code=EPIPE, syscall=write

I searched nearly all the internet for this error but no one provides a solution nor an explanation for what is going wrong.

Luckily the process "sometimes" shows a stack for the error. It points to lib/redis/index.ts:711 inside the ioredis codes. Which I have no idea what it does.

(stream || this.stream).write(command.toWritable());

https://github.com/luin/ioredis/blob/master/lib/redis/index.ts#L711

I found some issues on ioredis github repository mentioning some EPIPE error. But most of them were about error handling stuff and all marked as resolved.

I also found some general EPIPE errors on google(Most of them about socket.io which is not something I use.)

Wrap Up

What is wrong with this thing?

Answer 1

As no one wrote an answer at the end of bounty. I am writing my experience on solving the issue for the people who will get this error later.

Please note that this is not a canonical answer. But It is rather a workaround

I am starting with sharing what was happening.

We were trying to migrate from a Redis server hosting nearly 600 000 keys. Standard migration process was taking a lot of time for transfer that amount of keys from Redis to keyDB. So I came across a different solution.

Our KeyDB works on 2 Active-Active replica servers. I will provide the link to those are wondering how this system works.

https://medium.com/faun/failover-redis-like-cluster-from-two-masters-with-keydb-9ab8e806b66c

The solution was re-building up our Redis data using some MongoDB database aggregation and doing some batch operations on KeyDB.

Here is a simulation(Not exact as source. Also I did not tested for syntax errors)

const startPoint =
            (Number.parseInt(process.env.NODE_APP_INSTANCE) || 0) * 40000;
let skip = 0 + startPoint;
let limit = 1000;
let results = await SomeMongooseSchema.find({someQueries}).limit(1000).skip(skip);

let counter = 0;
while (results.length){
   if(counter > 39) break;
   for(const res of results){
      const item = {
         key: '',
         value: ''
      };
      // do some build ups on item
      ...
      // end n
      app.ioRedisClient.set(item.key, item.value);
   }
   counter++;
   skip = i * limit + startPoint;
   results = await SomeMongooseSchema.find({someQueries}).limit(limit).skip(skip);
}

Running this code on 16 processes using pm2 sets all the keys to keyDB in about 45 minutes. (compared to 4-5 hours)

   pm2 start app.js -i 16 

When we run the code on a Redis server. It works but giving the following error on KeyDB.

error:  message=write EPIPE, stack=Error: write EPIPE
./app-error-1.log:37:    at WriteWrap.onWriteComplete [as oncomplete] (internal/stream_base_commons.js:92:16), errno=EPIPE, code=EPIPE, syscall=write

Firstly I started by tuning the code by creating a transaction instead of setting every key separately. and setted a 1-second gap between each 1000 operation. the code changed as following.

const startPoint =
            (Number.parseInt(process.env.NODE_APP_INSTANCE) || 0) * 40000;
let skip = 0 + startPoint;
let limit = 1000;
let results = await SomeMongooseSchema.find({someQueries}).limit(1000).skip(skip);

const batch = app.ioredisClient.multi();
let counter = 0;
while (results.length){
   if(counter > 39) break;
   for(const res of results){
      const item = {
         key: '',
         value: ''
      };
      // do some build ups on item
      ...
      // end n
      batch.set(item.key, item.value);
   }
   counter++;
   await batch.exec();
   await sleep();
   skip = i * limit + startPoint;
   results = await SomeMongooseSchema.find({someQueries}).limit(limit).skip(skip);
}

This reduced the error rate as long as the operation time to 20 minutes. But the error was still persisting.

I suspected that this error may be due to some permission errors on the docker version. So I asked our server administrator to check and if possible remove the docker version and install from rpm repository.

https://download.keydb.dev/packages/rpm/centos7/x86_64/

Did that and it worked. All the errors vanished and successfully managed to migrate in 20 minutes.

I don't consider this as a real answer. But this should be useful for some experts to find out what was wrong.