Nested Cross-Validation
When a pipeline has hyperparameters (regularization strength, feature count, which channels), tuning them on the same data used to report performance leaks the test set and biases the score upward. Nested CV uses two loops: an inner CV over the training portion selects hyperparameters, and an outer CV — whose test folds the inner loop never saw — estimates generalization with those choices remade inside each outer fold. The reported number is the average over outer folds.
The cost is roughly the product of the two loop sizes in compute, and the payoff is an approximately unbiased estimate of the whole model-selection procedure rather than of a single lucky configuration. A flat single-loop CV that reports the best hyperparameter's own score is optimistically biased, sometimes severely so when many configurations are searched.
What is being estimated matters: nested CV evaluates the pipeline including its tuning, not a fixed model. If you then retrain the tuned pipeline on all data for deployment, the nested-CV score is your honest expectation of its performance — but the final selected hyperparameters may differ per outer fold, which is expected.