population-based training

Instead of training one agent, population-based training trains a whole crowd of them at the same time and lets them learn from each other's fortunes. Each member runs its own RL with its own hyperparameters; periodically the population takes stock, the weak copy a strong member's weights and settings, and those copied settings are then randomly perturbed to explore nearby. It blends ordinary learning (gradient steps within each agent) with evolution (selection and mutation across agents).

The first thing this buys you is automatic hyperparameter tuning during a single training run: there is no separate, expensive sweep, because the learning rate, entropy bonus, reward weights and so on are themselves evolving online as the well-performing combinations spread through the population. The second, and the reason it lives in this chapter, is that the population doubles as a pool of diverse opponents for self-play — a built-in supply of varied adversaries that resists the cycling and overfitting of single-line self-play.

Population-based training was central to AlphaStar's StarCraft II agents and to the capture-the-flag work in Quake, precisely because a varied population both tunes itself and produces robust, hard-to-exploit strategies. The cost is compute: you are running many agents in parallel, so it is far hungrier for resources than a single run, and the evolutionary part adds bookkeeping. Where compute is abundant, it is a remarkably effective way to get robustness and tuning for free-ish.