fault injection and testing the error paths
Imagine a fire-safety inspector who does not just admire the sprinklers - he actually lights a small controlled fire to check they switch on. You cannot trust an emergency system you have never seen react to an emergency. Fault injection is the same principle for software: deliberately make things fail on purpose, under controlled conditions, so you can watch your error-handling code actually run and confirm it does the right thing.
The reason this is necessary is uncomfortable but true: error paths are the least-tested code in almost every program. The happy path runs every time you use the program, so bugs there surface fast. But the code that runs when malloc() returns NULL, when write() fails because the disk is full, when a network read times out - that code might never run during normal development, so it can be wrong for years without anyone noticing. Fault injection forces those paths to execute. The techniques range from simple to sophisticated: a wrapper around malloc() that returns NULL on, say, every 100th call (to test out-of-memory handling); a test that closes a file descriptor early so the next read() fails; tools that intercept system calls and make them fail on demand; and at the extreme, chaos engineering, where you deliberately kill servers in a live distributed system to prove it survives. The goal is always the same: drive the program down the error branches it would otherwise almost never take, and check it cleans up, reports honestly, and does not corrupt state.
Why it matters: an error path that has never executed is, realistically, untested code, and untested error handling fails exactly when you most need it - during a real outage. Fault injection is how 'I wrote cleanup code' becomes 'I watched the cleanup code run correctly under failure'. The honest caveats: you cannot inject every possible fault, so it complements rather than replaces careful design and code review; injected faults must be controllable and reversible so your test harness does not leave real damage; and the test must check the outcome, not just that the program survived - a program that 'survived' a failed write() by silently discarding data has not passed, even though it did not crash.
A test-only malloc wrapper: void *test_malloc(size_t n) { if (--countdown == 0) return NULL; return malloc(n); } - set countdown so the third allocation fails, then run the function and confirm it cleans up the first two allocations and returns an error instead of crashing or leaking.
Forcing the third malloc to fail makes the rarely-run cleanup path actually execute and be checked.
Surviving an injected fault is not the same as passing: a program that 'kept running' by silently dropping data still failed. The test must verify the right outcome (cleanup happened, an honest error was returned), not merely that it did not crash.