explainable AI (XAI)
/ ik-SPLAY-nuh-bul AY-AY /
Explainable AI is the effort to make a model's reasoning understandable to a human — to turn the black box at least partly into a glass box. Many powerful models, especially deep neural networks, reach answers through millions of arithmetic steps no person can follow. XAI asks a humbler, practical question: can we offer an explanation that helps a human trust, check, or contest what the model did, even if we can't trace every neuron?
It splits into two broad approaches. Some models are interpretable by design — a short decision tree or a simple linear formula whose logic you can read off directly. Others are explained after the fact: you keep the opaque model but bolt on tools that produce a story about why it answered as it did, such as highlighting which inputs mattered most. The first is honest but often less accurate; the second is flexible but produces explanations that are approximations, not the model's actual inner workings.
XAI matters wherever a decision affects a person — a loan denied, a tumor flagged, a résumé filtered — because "the algorithm said so" is not an acceptable reason, and laws increasingly demand more. But here is the crucial honesty: an explanation that sounds plausible is not the same as a true account of the model's reasoning. Post-hoc explanations can be unstable, can disagree with each other, and can be gamed to look fair while the model misbehaves underneath. A good explanation builds appropriate trust; a glib one manufactures false trust.
A bank's model denies a loan. An explainability tool reports the top reasons: "high debt-to-income ratio" and "short credit history." That lets the applicant understand and contest the decision — and lets the bank check the model isn't secretly leaning on the applicant's zip code as a stand-in for race.
An explanation serves two masters: the affected person and the auditor checking for hidden bias.
Distinguish two things people lump together: explanation (a human-friendly story about one decision) and the model's actual mechanism (what really computed it). Most popular tools deliver the first while implying the second. A plausible explanation that doesn't faithfully reflect the model can be worse than none — it lends unearned confidence.