dictionary learning

Think of how a few dozen Lego brick types can build almost anything: every model, however complex, is just a sparse selection of bricks snapped together. Dictionary learning is the general idea of discovering such a set of basic building blocks — the dictionary — so that each complicated example in your data can be written as a small combination of a few of them. It is an old technique in signal processing, repurposed here to find the building blocks of a model's thoughts.

Formally, you have many activation vectors and you want to learn a dictionary of basis directions (called atoms) such that every activation is well-approximated by adding up just a few atoms. Two pressures fight: the reconstruction must be accurate, and the selection must be sparse (use few atoms per example). The sparsity is the crucial ingredient — it forces the atoms to become meaningful, reusable concepts rather than a vague blur, because the only way to rebuild thousands of examples from few-atom recipes is for the atoms themselves to capture real, recurring structure. A sparse autoencoder is the most popular modern way to actually carry out dictionary learning on a neural network.

For interpretability and safety, dictionary learning is the formal answer to superposition: if the model packed many features into few neurons by overlapping them, dictionary learning is the principled attempt to recover those original features as the dictionary's atoms. The honest caveat is that the right number of atoms and the right sparsity are choices, not facts, and the recovered dictionary is the method's best guess at the model's concepts — a useful, testable reconstruction, not a guaranteed match to whatever the model truly uses.