Redis replication: prevent split-brain (additional master) in case of network partition

Redis replication group can go multi-master on network partition, if old master restarts automatically. Sentinels will eventually fix it to become slave of new one, but that takes some time. If some application continues writing to old master, some data will be lost. To prevent this issue, write this line in every redis.conf:

slaveof 6739

And make redis.conf permissions strict, so that redis would not be able to rewrite it with CONFIG REWRITE. Every host after restart will become slave with broken master connection, multi-master will never happen. If all hosts were just booted and there is no master, you should choose one by giving it command slaveof no one. After some timeout sentinels will fix restarted host to replicate from new master.

And now, since you will never have second stand-alone master, you can use it with haproxy on each node to relieve clients from dealing with sentinels for every connection. I have found original idea in this post: haproxy blog - advanced redis health check


defaults REDIS
    mode tcp
    timeout connect  4s
    timeout server  30s
    timeout client  30s

frontend ft_redis
    bind name redis
    default_backend bk_redis

backend bk_redis
    option tcp-check
    tcp-check send PING\r\n
    tcp-check expect string +PONG
    tcp-check send info\ replication\r\n
    tcp-check expect string role:master
    tcp-check send QUIT\r\n
    tcp-check expect string +OK
    server R1 check inter 1s
    server R2 check inter 1s
    server R3 check inter 1s

License: This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License Share: share on reddit share on hacker news share on facebook share on twitter share on linkedin share on slashdot