Redis master/slave setup on Kubernetes throwing error: BRPOPLPUSH { ReplyError: MOVED 2651

Question

I'm using the excellent Redis based Bull.js as a job queue on Kubernetes.

It's configured as a cluster:

When Kubernetes restarts upon deployments, I run into this following error:

BRPOPLPUSH { ReplyError: MOVED 2651 <IP_ADDRESS>:6379
at parseError (/usr/src/app/node_modules/ioredis/node_modules/redis-parser/lib/parser.js:179:12)
at parseType (/usr/src/app/node_modules/ioredis/node_modules/redis-parser/lib/parser.js:302:14)
command:
{ name: 'brpoplpush',
args:
[ '{slack}:slack notifications:wait',
'{slack}:slack notifications:active',
'5' ] } }

Where <IP_ADDRESS> is, I think the cluster IP? I didn't configure this, but I'm trying to debug this. I want to know if I need to enable cluster mode for Bull.js or if this is a configuration issue outside of the Bull.js project?

Or is it a networking issue with K8s?

Would enabling: https://github.com/OptimalBits/bull#cluster-support be the solution? Is this the right approach?

Here is my code:

import Queue from 'bull';
import config from 'config';
import { run as slackRun } from './tasks/send-slack-message';
import { run as emailRun } from './tasks/send-email';

const redisConfig = {
  redis: {
    host: config.redis.host,
    port: config.redis.port
  }
};

const slackQueue = new Queue('slack notifications', { ...redisConfig, ...{ prefix: '{slack}' } });
const emailQueue = new Queue('email notifications', { ...redisConfig, ...{ prefix: '{email}' } });

slackQueue.process(slackRun);
emailQueue.process(emailRun);

emailQueue.on('completed', (job, result) => {
  job.remove();
});

export { emailQueue, slackQueue };

import { emailQueue, slackQueue } from 'worker/worker';

const queueOptions = {
  attempts: 2,
  removeOnComplete: true,
  backoff: {
    type: 'exponential',
    delay: 60 * 1000
  }
};

emailQueue.add(
  {
    params: {
      from: email,
      fromname: name,
      text: body
    }
  },
  queueOptions
);
slackQueue.add(
  {
    channelId: SLACK_CHANNELS.FEEDBACK,
    attachments: [
      {
        text: req.body.body
      }
    ]
  },
  queueOptions
);

This is the configmap:

Name:         redis-cluster-config
Namespace:    default
Labels:       <none>
Annotations:  <none>

Data
====
update-node.sh:
----
#!/bin/sh
REDIS_NODES="/data/nodes.conf"
sed -i -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/" ${REDIS_NODES}
exec "$@"

redis.conf:
----
cluster-enabled yes
cluster-require-full-coverage no
cluster-node-timeout 15000
cluster-config-file nodes.conf
cluster-migration-barrier 1
appendonly yes
# Other cluster members need to be able to connect
protected-mode no

Events:  <none>

Answer 1

Hitobat is right though.

If this doesn't help:

const Redis = require('ioredis');
...
const ioCluster = new Redis.Cluster([redisConfig.redis]);
const slackQueue = new Queue('slack notifications', {
  prefix: '{slack}' ,
  createClient: () => ioCluster
});
const emailQueue = new Queue('email notifications', {
  prefix: '{email}' ,
  createClient: () => ioCluster
});

I would go without ioredis or try to downgrade redis engine to 4.x.