Finding the Wrong Labels: Confident Learning and Weak Supervision

Noise is not random

Real label noise is structured: annotators confuse specific pairs of classes, ambiguous instances get coin-flipped, and rare categories are systematically collapsed into common ones. Modeling noise as a uniform random flip is convenient and almost always wrong. The right object is the joint distribution over (true label, observed label) — a confusion-like matrix whose off-diagonal mass tells you exactly which mistakes happen and how often.

Confident learning estimates that joint without trusting the noisy labels too much. The key idea is a per-class confidence threshold: an example is counted as belonging to class j only when the model's predicted probability for j exceeds the average self-confidence the model assigns to true members of j. This thresholding cancels much of the model's miscalibration and class-frequency bias, so the off-diagonal counts approximate genuine mislabelings rather than ordinary mistakes.

t_j=\frac{1}{\lvert X_{\tilde{y}=j}\rvert}\sum_{x\in X_{\tilde{y}=j}}\hat{p}\!\left(\tilde{y}=j;\,x,\theta\right)

The per-class self-confidence threshold: class j's threshold is the average predicted probability of j over the examples labeled j.

The confident-learning recipe

Get out-of-sample predicted probabilities for every example via cross-validation, so no point is scored by a model that trained on it.
Compute each class's self-confidence threshold as the mean predicted probability over examples labeled that class.
Build the confident joint by counting, for each labeled class, how many examples confidently belong to each other class.
Rank suspected errors by margin (predicted prob of the off-diagonal class minus that of the given label) and route the top of the list to re-annotation.

psx = crossval_predict_proba(X, y)          # out-of-sample probs
thr = [psx[y==c, c].mean() for c in classes]  # per-class confidence
joint = confident_joint(psx, y, thr)         # noisy x true counts
suspects = rank_by_margin(psx, y)            # most-likely mislabeled first

Confident learning in four lines: cross-validated probabilities, per-class thresholds, the confident joint, then a ranked error list.

When labels don't exist yet

Confident learning cleans labels you already have; programmatic labeling creates them when hand annotation cannot scale. You write many noisy labeling functions — heuristics, regex rules, distant-supervision lookups, or prompts to a large model — each of which votes on a subset of examples and abstains elsewhere. These are classic weak supervision sources: individually unreliable, often correlated, but collectively informative.

A label model then learns each source's accuracy and their correlation structure without ground truth, by fitting the agreements and disagreements among the functions. It outputs a probabilistic label per example, which you train on with a noise-aware loss. The art is in the sources: cover the space, keep their errors diverse, and watch for a dominant source that quietly imposes its bias on everything downstream.

P_\theta(y,\lambda)=\frac{1}{Z_\theta}\exp\!\left(\sum_{j=1}^{m}\theta_j\,\mathbb{1}\!\left[\lambda_j=y\right]\right)

The weak-supervision label model: it weights each labeling function by a learned accuracy θ_j, rewarding agreement with the latent true label y — no ground truth required.

Tying it back to quality

Both techniques feed the same dashboard. The estimated joint from confident learning is a data-quality metric — its off-diagonal mass is your label-noise rate, broken down by class. The label model's accuracies tell you which weak sources to trust. Under class imbalance and a heavy tail, report these per class, never just in aggregate, or a clean head will hide a broken tail.