hyperparameter tuning
/ HY-per-puh-RAM-uh-ter TOON-ing /
Hyperparameters are the settings you choose before training starts — the learning rate, the batch size, how many layers, how strong the weight decay — as opposed to the weights, which the model learns by itself. Hyperparameter tuning is the search for a good combination of these settings. Think of baking: the recipe's oven temperature and bake time are hyperparameters you must dial in, while the dough's final texture is what "learns" itself once those are fixed.
Because you usually can't calculate the best settings, you experiment. Grid search tries every combination on a predefined grid of values — thorough but expensive, since the number of combinations explodes as you add more knobs. Random search samples combinations at random and, perhaps surprisingly, often finds good settings faster than a grid, because it doesn't waste effort exhaustively varying knobs that turn out not to matter. More advanced methods (like Bayesian optimization) use earlier results to decide what to try next.
Each candidate is judged on a validation set, never the test set — that's reserved for the final, honest estimate. The realistic picture is that tuning is costly: every trial is a full (or partial) training run, so people lean on good defaults, tune the few hyperparameters that matter most (learning rate first), and budget their search. A subtle trap is tuning so hard against the validation set that you overfit to it, which is why a separate, untouched test set is non-negotiable.
Tuning two knobs — learning rate (try 0.1, 0.01, 0.001) and batch size (try 32, 128). Grid search runs all 3×2 = 6 combinations. Random search might instead sample 6 random (rate, batch) pairs from continuous ranges, often landing nearer the true best because it explores more distinct learning-rate values — the knob that matters most here.
Random search often beats grid search per dollar of compute.
Always pick hyperparameters on the validation set and report final numbers on a held-out test set. Tuning against the test set quietly leaks information and inflates your results — a common, costly form of self-deception.