redundancy
If you only carry one house key and lose it, you are locked out. If you keep a spare under a flowerpot, losing one key is a shrug, not a crisis. Redundancy is exactly that idea: keep more than one copy of anything you cannot afford to lose, so the failure of any single copy does not stop you. In a warehouse-scale computer this is not a luxury but a survival requirement, because at the scale of tens of thousands of machines, components fail constantly — the working assumption is that some servers, disks, switches, and power supplies die every single day. Redundancy is how a system built from unreliable parts becomes a reliable whole.
Concretely, redundancy means storing each piece of data on several machines (replication) and running each service on several servers, ideally spread across different racks, power circuits, and even buildings, so that no single failure takes out all the copies at once. If a server holding one replica dies, requests are simply routed to another replica that has the same data; if a whole rack loses power, the copies in other racks carry on. The system continuously watches for failures and, when a copy is lost, makes a fresh copy somewhere healthy to restore the safety margin. The price is obvious: storing three copies of everything costs three times the storage, and running spare capacity costs money even when nothing has failed.
Redundancy is the foundation of dependability at scale, and it changes how you think about hardware. Because you assume failure is routine, you do not need each machine to be ultra-reliable and expensive — you can use cheap commodity parts and let redundancy across many of them deliver the reliability, the way a flock survives even though individual birds do not. This is a deep WSC philosophy: reliability is engineered at the system level through replication and fast recovery, not bought at the component level through gold-plated hardware.
An honest caveat that bites real systems: redundancy only protects you against failures that are independent. Three copies are useless if all three sit in the same rack and the rack loses power, or if a single bug, bad deploy, or correlated event takes them all out together. So real designs work hard to make replicas fail independently — different racks, different power, different software versions during a rollout — and they must also be tested, because an untested failover path is a copy you only discover is broken at the worst possible moment.
A file system stores every block three times, on three servers in three different racks. A disk failure loses one replica; reads switch to either of the other two, and the system quietly rebuilds a third copy on a healthy disk. The user never notices a thing — even though, across the fleet, disks are failing all day.
Three replicas across three racks: any single failure is invisible, and the lost copy is rebuilt automatically.
Redundancy only helps against independent failures. Three replicas in one rack, or copies killed by one bad deploy, fail together. And an untested failover path is a copy you only learn is broken during the outage.