SHAP

SHAP is a method for splitting a single prediction into fair credit for each input feature, borrowing a recipe from game theory. Imagine a team wins a prize and you must divide it fairly among players who contributed unequally. The Shapley value, from economics, answers exactly this: it considers every possible order in which players could have joined and averages each one's marginal contribution. SHAP treats the model's prediction as the prize and each feature as a player.

Concretely, for one specific case — this patient, this loan — SHAP tells you how much each feature pushed the prediction up or down relative to a baseline average. The values add up exactly to the gap between the model's prediction and that baseline, which is why it's called additive: the explanation perfectly accounts for the output, no leftover. This makes SHAP popular for both local explanations (why this one decision?) and, by aggregating, global ones (what does the model rely on overall?).

SHAP's appeal is that it rests on solid theory and gives consistent, additive answers, unlike ad-hoc methods. Its honest limits: exact computation is exponentially expensive, so real tools use approximations whose quality varies. It assumes you can sensibly remove features, which strains when features are correlated. And — the big one — SHAP explains what the model does, not whether the model is right or whether a feature truly causes the outcome. A confident SHAP chart for a biased model just explains the bias beautifully.