Designing for Constant Failure

At scale, 'broken' is the steady state

From the first guide in this rung you already treat the whole warehouse-scale computer as a single machine, and from the last two you know the work is sprayed across thousands of servers and the network that stitches them together. Now confront the consequence that makes datacenter design feel alien: at this scale, something is always broken. A disk dies, a memory stick flips bits, a power supply quits, a switch reboots, a whole rack loses power — not occasionally, but continuously. The right mental model is not 'parts fail sometimes' but 'parts are failing right now, somewhere, every hour of every day.'

The arithmetic is brutal but simple. Suppose one commodity server has a mean time between failures of 3 years — pretty reliable for a single box. Stack 50,000 of them and the fleet's collective failure rate is 50,000 times higher, so on average a machine drops out roughly every 30 minutes, all day, forever. You cannot fix this by buying nicer hardware: doubling each server's reliability only halves the rate, and you have 50,000 of them. Reliability that comes from making each part perfect simply does not scale; it has to come from somewhere else.

Fault, error, failure: getting the words right

To design for constant trouble, you first need precise words for what 'trouble' even means. The standard chain from the dependability literature has three links. A fault is a defect — a stuck bit, a cracked solder joint, a cosmic-ray strike. A fault may lie dormant for hours. When it actually corrupts some state or produces a wrong value, that is an error. And when that error becomes visible to the outside — a user sees a wrong answer, a request times out — that is a failure. The whole craft of reliable systems is intercepting the chain before fault becomes failure.

  fault            error               failure
  (defect)  --->   (wrong state)  ---> (visible to user)

  cosmic ray   ->  flipped bit    ->   crash / wrong reply
     |               |                    |
  ECC catches it  replica masks it   ...nothing reaches the user
     V               V                    V
  chain BROKEN     chain BROKEN        no failure observed

Why fuss over the vocabulary? Because each link is broken by a different mechanism, and confusing them leads to fixing the wrong thing. You stop a fault from becoming an error at the hardware level — for example, error-correcting code memory that detects and repairs a single flipped bit before any program ever reads it. You stop an error from becoming a failure at the system level — for example, by keeping a second copy of the data on another machine so that when one server returns garbage or nothing at all, the request is simply served from the replica. Same goal, two completely different layers.

Redundancy: paying for spares so failure is boring

The master technique is redundancy: have more of everything than you strictly need, so the loss of any one piece is absorbed without anyone noticing. You met a small version of this idea in the storage rung as RAID, where data is spread across disks with parity so one dead disk loses nothing. The warehouse takes the same bet and applies it everywhere — to disks, to servers, to power feeds, to network paths, to whole datacenters in different cities. Spare capacity is not waste; it is the premium you pay so that a constant drizzle of failures never becomes a storm.

Concretely, the standard pattern for data is replication: keep three copies of every chunk, on three different machines, ideally in three different racks so that a single rack power loss cannot take all three. A read can be served by any copy; a write must reach all of them (or a safe majority). When a machine dies, the system notices its copies have gone missing and quietly re-replicates them onto healthy machines to restore the count of three — a background self-healing process that runs forever. From the outside, a disk dying just means a number drifting back to three.

Graceful degradation: bending instead of breaking

Redundancy hides small failures. But sometimes too much breaks at once — a whole rack drops, a region of the network partitions, a surge of traffic outstrips capacity. The design goal then is graceful degradation: the service gets worse a little, in proportion to the damage, instead of falling over entirely. A search engine that loses a tenth of its index servers should return slightly less complete results a hair slower, not show an error page. The system bends; it does not snap.

This connects straight back to the tail-latency story from the previous guide. A request that fans out to a thousand servers is hostage to the slowest of them, so a failing or overloaded machine does not just lose its own work — it can drag the whole response into the long tail. Graceful degradation says: rather than wait forever for a sick replica, set a deadline, and when it passes, answer with what you have. Returning a 95%-complete result in 100 milliseconds beats returning a perfect result in 10 seconds that the user has already abandoned.

1. A request fans out to many replicas, each holding part of the answer.
2. Each shard has a deadline; a healthy replica answers well within it.
3. If a replica is dead or too slow, the coordinator does not wait — it either skips that shard or asks a backup copy.
4. The user gets a slightly-less-complete answer, on time, and usually never knows anything was wrong.

Honest limits: what redundancy cannot save you from

It is tempting to believe enough replicas make a service immortal. They do not, and the failure modes redundancy misses are exactly the ones that cause the famous multi-hour outages. Redundancy assumes failures are independent — that two copies on two machines will not die for the same reason at the same time. Correlated failures break that assumption with a vengeance: a bad software push rolled out to every server at once, a configuration change that poisons the whole fleet, a power event that takes a whole datacenter, a leap-second bug that crashes every machine in the same second. Three copies of the same buggy binary fail three times over.

So the honest picture is layered, not absolute. Per-component tricks like ECC mop up the constant drizzle of bit flips. Replication and graceful degradation absorb the steady rain of independent machine and rack deaths. But correlated failures need a different defence entirely: rolling out changes slowly to a few machines first (canarying), spreading copies across independent failure domains, and rehearsing disaster so the recovery path is known to work. No amount of redundancy substitutes for not breaking everything at once.

Step back and the philosophy is clear, and it is the thread tying this whole rung together. You do not chase a warehouse-scale computer that never fails — at a hundred thousand parts that is a fantasy. You build a service that keeps working while its parts fail, continuously, underneath it. Failure stops being an emergency and becomes ordinary weather you have already dressed for. With that mindset in place, the last guide can finally ask the question every operator cares about: given that you must buy all this redundancy and power, what does it actually cost?