leader election
Many distributed tasks are far simpler if exactly one node is in charge — one coordinator to hand out locks, one node to decide the order of updates, one place that breaks ties. But you cannot just appoint a permanent boss, because in a distributed system the boss can crash at any time (partial failure). Leader election is the procedure by which a group of equal nodes, after the current leader has failed or at startup, agree on exactly one of them to be the new leader — automatically, despite the absence of any central authority.
A simple, classic example is the Bully algorithm. Every node has a unique id number, and the rule is "highest id wins". When a node notices the leader is unresponsive, it sends an election message to all nodes with higher ids. If none of them answers (they are down too), it declares itself leader and announces this to everyone. If a higher-id node does answer, that node takes over the election in the same way, eventually leaving the highest live id as the winner — it bullies the lower ones aside. Other algorithms (like the ring algorithm) circulate a message around a logical ring instead. Whatever the method, the goal is the same: end with everyone agreeing on one leader.
Why it matters, and the subtle danger: leader election is the recovery mechanism that keeps coordinator-based designs alive when the coordinator dies — it appears inside consensus systems like Raft, inside cluster managers, inside distributed databases. The classic pitfall is split-brain: if the network partitions, two halves might each think the old leader is gone and elect their own, ending up with two leaders who disagree and corrupt the shared state. This is why robust election usually requires a majority (a quorum) to agree on the new leader — a minority partition simply cannot elect one, which keeps the system safe at the cost of that minority's availability.
Five database replicas have ids 1 to 5; node 5 is the leader. Node 5 crashes. Node 3 notices, sends election messages to 4 and 5; 4 replies and takes over, gets no reply from 5, so node 4 declares itself leader and tells everyone. The group now has a single agreed leader again, with no human intervention.
After the leader dies, the survivors automatically agree on a new one. Beware electing two at once (split-brain).
The dangerous failure is split-brain: a network partition that lets two sub-groups each elect a leader. Requiring a majority quorum prevents this — a minority can never form a leader, so there is never more than one.