Redis Fail Over

Question

We have configured the Redis server with one master and two slaves. If my master fails, how can we handle the failover without restarting the Redis server.

Answer 1

Update:

Today, I would recommend checking out redis-sentinel, a tool by Redis' author antirez for monitoring and automatic failover.

Original reply:

Check the SLAVEOF command: http://redis.io/commands/slaveof

When you discover that your master fails, issue a SLAVEOF NO ONE on one of your slaves to promote it to master. Then point your other slave to it's new master. See also "Upgrading or restarting a Redis instance without downtime": http://redis.io/topics/admin

For managing configuration files you could do something along these lines (caution: Not tested, meant as an example). The example below assumes two configuration files for each server (/etc/redis/server1.master.conf, /etc/redis/server1.slave.conf, etc), one having that server as a slave of some predefined master:

#!/bin/sh  master() {     server_name=$1     redis-cli slaveof no one     ln -sf /etc/redis/$server_name.master.conf /etc/redis/$server_name.conf }  # Usage: slave(server1 server2 6379) slave() {     server_name=$1     master=$2     master_port=$3     redis-cli slaveof $master $master_port     ln -sf /etc/redis/$server_name.slave.conf /etc/redis/$server_name.conf } 

Instead of having the predefined configuration files, you could edit them on the fly with e.g. sed. Basically, you would make sure to always have a slaveof stanza in the configuration files, either pointing to a master or slaveof no one. Then rewrite the configuration using sed (again, not tested, just meant as food for thought):

#!/bin/sh  master() {     server_name=$1     config=$server_name.conf     redis-cli slaveof no one     sed -i "s/^slaveof.*/slaveof no one/" $config }  # Usage: slave(server1 server2 6379) slave() {     server_name=$1     config=$server_name.conf     master=$2     master_port=$3     redis-cli slaveof $master $master_port     sed -i "s/^slaveof.*/slaveof $master $master_port/" $config } 

Answer 2

I would recommend to look into Redis Cluster (version 3.2 as latest stable today). Cluster this is a new approach, no sentinels any more. Fail over principle is the same, slave promotes to master in case of master is down plus new more functionality including sharding logic supported by Redis. Application just needs to connect to cluster having set of nodes, that's it.

If we are talking about general Fail Over, please be aware, Redis Cluster does not guarantee strong consistency.

Please find snippet from: http://redis.io/topics/cluster-tutorial

"Redis Cluster is not able to guarantee strong consistency. In practical terms this means that under certain conditions it is possible that Redis Cluster will lose writes that were acknowledged by the system to the client.

The first reason why Redis Cluster can lose writes is because it uses asynchronous replication.

There is another notable scenario where Redis Cluster will lose writes, that happens during a network partition where a client is isolated with a minority of instances including at least a master."