confident learning

Confident learning is a way to find the mislabeled examples in your dataset without ever knowing the true labels — by letting the model's own confident mistakes betray them. The intuition: train a classifier with cross-validation so every example gets an out-of-sample predicted probability, then look for examples the model is confidently sure belong to a class other than the one written on their label. Those are your prime suspects for label errors.

Formally it estimates the joint distribution between the noisy observed labels and the latent true labels. For each example you have predicted probabilities; you set per-class confidence thresholds (the average self-confidence for each class) and count an example into a confusion cell only when its predicted probability for some class exceeds that class's threshold. Aggregating these counts and calibrating gives an estimate of the noisy-vs-true joint matrix, from which you rank examples by estimated probability of being mislabeled and prune or relabel the worst. Crucially it is model-agnostic and assumes class-conditional noise rather than uniform flipping.

It scales to large datasets and underpins tools like cleanlab, but it inherits the model's blind spots: a systematic bias shared by the classifier and the labelers will pass undetected, and aggressive pruning can delete genuinely hard but correct examples.