Split Brain Scenario

Giriş
Master ve Slave düğümler arasında ağ (network) problemi olması. Açıklaması şöyle.

This is a classical problem in any distributed system where each node thinks the other nodes are down, while in reality, only the network communication between the nodes is broken. This scenario is more commonly known as a split-brain scenario, and if not handled properly, can lead to more than one node claiming to be a master ... which, in turn, leads to data inconsistencies and corruption.

Yıllar önce buna benzer bir problemle karşılaşmıştım ve o zaman bir problemin ismini bilmediğim için düzgün araştırma da yapamamıştım. Problemin ismini öğrenince not almak istedim.

Çözüm

3 tane farklı çözüm var. Açıklaması şöyle

1. Third-party actor (or witness) — an independent application outside of the cluster that can check the availability of all the nodes. Several types of applications can be used for this purpose: load-balancer, Zookeeper, or some other dedicated cluster software. And there are a variety of scenarios of how exactly this application protects the cluster from the split-brain problem. It can check the availability of nodes with heartbeats or register the nodes inside the application, allowing only one primary node to be registered.

2. Consensus — the decision about the current leader is based on the nodes’ vote. To promote one of the replicas to be the primary one, it should get a majority of votes, or quorum[3]. This approach is used in MongoDB replica sets. Another example is Hazelcast, which uses the quorum approach for write operations to protect itself from split-brain [4]. When an operation can’t be performed on the sufficient number of cluster members, it raises an exception.

3. Generation numbers — there is a generation number available across the cluster, which monotonically increases every time the leader is changed. All the nodes accept only actions performed using the current value of this number. When the old leader is disconnected from other nodes, it will keep the old generation number and won’t be able to apply changes anymore.

Split Brain ve Distributed Lock

Açıklaması şöyle.

But in general, when we are talking about locks, it is better to have only one node in charge of a particular lock in order to prevent the split-brain issue.