programmatic labeling

Instead of paying humans to label every example, programmatic labeling lets domain experts write rules — keyword heuristics, regexes, ontology lookups, calls to other models — that each guess a label for some fraction of the data. Each rule is a noisy, partial labeling function: it may be wrong, may abstain, and may correlate with other rules. The goal is to combine many such weak, cheap signals into accurate training labels at scale, turning expert intuition into supervision without manual annotation of each row.

The core machinery is a label model that estimates each labeling function's accuracy and their correlation structure from their agreements and disagreements alone — without ground truth — typically via a generative model over the latent true label and the observed votes, solved by matrix completion or method-of-moments on the covariance of function outputs. It then outputs a probabilistic (soft) label per example by weighting each function by its estimated reliability. A downstream discriminative model trained on these soft labels generalizes beyond the rules' coverage, learning features the heuristics never named.

The Snorkel line of work popularized this. The caveats: if labeling functions share a correlated blind spot the label model can be confidently wrong, and accuracies are only identifiable under structural assumptions about dependencies.