feature importance
/ FEE-cher im-POR-tunss /
Feature importance is a ranking of which inputs a model leans on most. If a model predicts house prices from square footage, neighborhood, age, and paint color, feature importance tells you that square footage and neighborhood drive the predictions while paint color barely registers. It is the most basic question of interpretability: of everything the model could pay attention to, what actually moves its answers?
There are two honest ways to ask this, and they answer different questions. One measures how much the model relies on each feature internally — how often a decision tree splits on it, say. The other, permutation importance, is more model-agnostic: scramble one feature's values across the dataset and see how much accuracy drops. A big drop means the model truly needed that feature. A simple, intuitive idea — but the devil is in correlation.
Feature importance matters because it is the first place teams catch a model cheating. If "patient ID number" turns out to be a top predictor of disease, something is wrong: the model has latched onto a leak rather than medicine. But the caveat is sharp. When features are correlated, importance gets smeared confusingly across them, and different methods can rank the same features in completely different orders. Importance tells you what the model used — never assume it tells you what actually causes the outcome.
A model predicting hospital readmission lists "number of prior visits" and "age" as its top features — sensible. But "patient came from Building 3" also ranks high. Investigation reveals Building 3 houses the oncology ward; the model had quietly discovered that cancer patients get readmitted more, a real but unintended shortcut worth knowing about.
An unexpected high-importance feature is often a clue, not a verdict — investigate before trusting it.
Importance is about the model, not the world. A feature can rank high because the model uses it, even if it has no causal effect on the outcome — and a genuinely causal feature can rank low if a correlated one steals its credit. Never read a feature-importance chart as a causal map.